FULLTEXT01.Pdf

Abstract Kinect-based Music Application for Children with Severe Physical Disabilities I Made Satrya Rudana Teknisk- naturvetenskaplig fakultet UTH-enheten Based on initial interviews with music teachers at Årstra Special School, Uppsala, it was found that each child in a music playing Besöksadress: session has different preferences of type and sound of a music Ångströmlaboratoriet Lägerhyddsvägen 1 instrument. However, most of them have combined cognitive and physical Hus 4, Plan 0 impairments, preventing them from playing the instrument that they might like. Postadress: Box 536 751 21 Uppsala Starting from this idea, we developed a music application using virtual instruments, so that various types of instruments and sound can be used Telefon: during a single music playing session. As an input device, we used a 018 – 471 30 03 Kinect sensor developed by Microsoft, i.e., a camera based sensor that Telefax: detects human gestures. Our application used this Kinect sensor 018 – 471 30 00 capability to allow users to control and play the sound by just moving their arms in the air. Hemsida: http://www.teknat.uu.se/student Our study has shown promising results of this applicatio, such as the positive response from the participant towards the application and the ability to change the sound of an instrument to match the participant's preference easily. However there are still some things to consider before releasing it as a consumer product, for instance better calibration and accuracy. Handledare: Lars Oestreicher Ämnesgranskare: Justin Pearson Examinator: Anders Jansson IT 17011 Contents 1 Introduction 4 1.1 Background..................................... 4 1.2 ResearchQuestions................................. 6 1.3 ResearchGoals ................................... 6 1.4 RelatedWorks ................................... 6 2 Background Theory 8 2.1 MIDI......................................... 8 2.1.1 MIDI Messages . 9 2.1.2 Virtual Instrument . 10 2.2 Kinect........................................ 11 3 Method 13 3.1 PilotStudy ..................................... 13 3.2 Target group . 13 3.3 InstrumentPlayingSession . 13 3.4 DataGathering................................... 14 3.5 Software and Hardware Requirements . 14 3.5.1 Digital Audio Workstation . 14 3.5.2 Processing3 ................................ 15 3.5.3 Microsoft Windows 10 . 16 4 Implementation 17 4.1 Kinect-based Music Application . 17 1 4.1.1 Hardware Requirements . 17 4.1.2 Skeletal Tracking . 19 4.1.3 Joint Angle Calculation . 19 4.1.4 Joint Angle to MIDI Message . 20 4.1.5 User Interface . 20 5 Result 23 5.1 Tracking....................................... 23 5.2 Participant Response . 25 6 Discussion and Future Research 27 6.1 Calibration and Accuracy . 27 6.2 Portability, Ease of Use, and Price . 27 6.3 Support ....................................... 28 6.4 ConnectedSystem ................................. 28 6.5 Evaluation Method . 28 7 Conclusion 29 References 30 Appendices 32 2 List of Figures 1 Roland Jupiter-8, an 8-voice polyphonic analogue synthesizer manufactured by Roland in 1981 without MIDI capabilities (Roland, 2014) . 8 2 Roland Midi Processing Unit MPU-401 (Jim, 2014) . 9 3 KinectV2 ..................................... 12 4 FruityLoops .................................... 15 5 Ableton Live . 15 6 Processing ..................................... 16 7 Kinect-based Music Application Process Diagram . 18 8 Upper Body Joints and Pelvis Joints . 19 9 User Interface of the Application . 22 10 SuccessTracking .................................. 23 11 False Detection of Arm . 25 3 1 Introduction This master thesis is conducted in IT Department of Uppsala University, Sweden as a part of a bigger research project led by Lars Oestreicher called Muminprojektet. Muminprojektet or the Mumin Project (Oestreicher, 2016) combines the capabilities of current available technology and music as a music instrument for people with physical disabilities. The part of the project explored in this thesis focused on an active music participation activity for the patient while this thesis focused on the possibility of using Kinect sensor as a music instrument in a Kinect-based music application. 1.1 Background Based on the interview with one of the teachers in Arsta˚ Special School, it was found that each child has a di↵erent preference regarding to the instrument type, when they are having a music therapy session at school. This observation proved to be a very important point in the music playing sessions especially for children with physical disabilities, because a research showed that children’s limitation of motor skills in certain leisure activities could lead to both disappointment and frustration (Kanagasabai, Mulligan, Mirfin-Veitch, & Hale, 2014). There are several instruments that are available in the school at the moment for the Mumin project. There are traditional instruments, such as: piano, snare drums, xylophones, one stringed bass guitars, and also more alternative instruments, such as: drum pads, electric harp, Theremini and other electronic instruments. The instrumetns are always adapted to the children’s abilities, so, for example, some of the children have very limited abilities to move their arm, and could only move their fingers at most. Thus they would prefer an instrument that is easy to play and suits their reduced motor skill. For instance, based on our initial interview in Arsta˚ Special School, children who only could control their fingers prefer an instrument such as Theremin1. Other children, who had more flexible and stronger arms, preferred to play an instrument that has tactile feedback for example a piano or drum pad. We tried to avoid the use of term therapy in this application development because we did not evaluate this application clinically and it has thus not been proven to be a therapy tool. Still we have tried to develop an application that can be used in a music playing session in Arsta˚ special school. We still suspect from the ongoing research work, that this application has a great potential to be used in a physical therapy if a certain 1Theremin is an instrument that can be played without touching. Small hand gestures around the antenna is used to control the pitch and volume (World, 2005). 4 medical guidelines taken into consideration when developing this application. The development of hardware nowadays has caused technical issues in music pro- duction — like latency or response time, memory capacity limitation, hard disk speed limitation, and processing power — slowly being reduced to minimum, even allowing a virtual instrument to be played live by a professional musician on stage. The advantage of virtual instrument compared to an normal instrument is the virtual instrument can produce the sound of any music instrument while can be played in various way. This is essential when we design a music instrument for children with various physical disabilities. Children who enjoy the sound of an acoustic guitar for instance, might not be able to play a regular acoustic guitar because they have too weak arms to hold the guitar. With the help of technology, music therapist could provide a music instrument with buttons that is connected to a computer so the children can play the sound of guitar easily only by pushing di↵erent buttons. Research in computer vision, virtual reality, and augmented reality has now given opportunity for the creation of applications with potential benefits for people with disabilities (Gonzlez-Ortega, Daz-Pernas, Martnez-Zarzuela, & Antn-Rodrguez, 2014). This includes the use of Kinect sensor and compatible applications to motivate people with physical disability to perform several activities as physical therapy. For instance, a Kinect-based therapy application made by Reflexion Health called Vera system, has a United States Food and Drug Association Premarket approval, which means that the application is soon to be on the market (Microsoft, 2015). Further research suggested future improvements of adding entertaining and amusing elements this type of applications that uses Kinect sensor as input device (Chang, Chen, & Huang, 2011). Starting from that, this research tried to replicate the work done in previous researchs (Gonzlez-Ortega et al., 2014), (Microsoft, 2015), and (Chang et al., 2011) with di↵erent type of application implementation. This research would explore the capabilities of Kinect to be used as an input device and connect it to a virtual instrument program. The users were then encouraged to move their body and trigger di↵erent sound’s volume and pitch. However, there are still some issues concerning to the therapy session itself. It has been shown that therapy sessions in a therapy center were very time consuming (Christy, Chapman, & Murphy, 2012). Transportation is needed to transport the patient from home to the therapy center and back. A home therapy session with family member has then been considered as an alternative. Funding or financial issues can also be a challenge because most of therapy session required trained therapist and tools to support the therapy (Cada & O’Shea, 2008). It will cost a large amount of money to pay for every therapy session. Also, parents that have children with disabilities pointed out the availability of trained therapists still being an issue (Cada & O’Shea, 2008). By combining the capabilities of the Kinect 5 sensor and a computer, this research aims to build a working system that considers such important factors for the target user groups as portability,

Load more