technologies Article Choreographic Pattern Analysis from Heterogeneous Motion Capture Systems Using Dynamic Time Warping Ioannis Rallis 1,*,†, Eftychios Protopapadakis 1,† , Athanasios Voulodimos 2,† , Nikolaos Doulamis 1,† , Anastasios Doulamis 1,† and Georgios Bardis 2,† 1 School of Rural and Surveying Engineering, National Technical University of Athens, 9, Heroon Polytechniou Str., 15773 Zografou, Greece 2 Department of Informatics and Computer Engineering, University of West Attica, Agiou Spyridonos Str., 12243 Egaleo, Greece * Correspondence: [email protected]; Tel.: +30-210-772-2664 † These authors contributed equally to this work. Received: 25 July 2019; Accepted: 14 August 2019; Published: 16 August 2019 Abstract: The convention for the safeguarding of Intangible Cultural Heritage (ICH) by UNESCO highlights the equal importance of intangible elements of cultural heritage to tangible ones. One of the most important domains of ICH is folkloric dances. A dance choreography is a time-varying 3D process (4D modelling), which includes dynamic co-interactions among different actors, emotional and style attributes, and supplementary elements, such as music tempo and costumes. Presently, research focuses on the use of depth acquisition sensors, to handle kinesiology issues. The extraction of skeleton data, in real time, contains a significant amount of information (data and metadata), allowing for various choreography-based analytics. In this paper, a trajectory interpretation method for Greek folkloric dances is presented. We focus on matching trajectories’ patterns, existing in a choreographic database, to new ones originating from different sensor types such as VICON and Kinect II. Then, a Dynamic Time Warping (DTW) algorithm is proposed to find out similarities/dissimilarities among the choreographic trajectories. The goal is to evaluate the performance of the low-cost Kinect II sensor for dance choreography compared to the accurate but of high-cost VICON-based choreographies. Experimental results on real-life dances are carried out to show the effectiveness of the proposed DTW methodology and the ability of Kinect II to localize dances in 3D space. Keywords: choreographic trajectory analysis; dance movement; Kinect; VICON; motion pattern matching; dynamic time warping 1. Introduction Intangible Cultural Heritage (ICH) is a prominent element of people’s cultural identity as well as a significant aspect for growth and sustainability [1]. The expression of identity through Intangible Cultural Heritage takes many forms, among which folkloric dances hold a central position [2]. It is reasonable to consider that analyzing choreographic sequences is essentially a multidimensional modelling problem, given that both temporal and spatial factors should be taken into account. Research has been published in the literature pertaining to ICH preservation which focuses on the time element [3–6]. Typical preservation acts include digitization, modelling, and documentation. Another important factor in preserving any type of performing arts, would be the development of an interactive framework that enhances the learning procedure of folklore dances. The recent advances in depth sensors, which have concluded to the development of low-cost 3D capturing systems, such as Microsoft Kinect [7] or Intel RealSense [8], permit easy capturing of human skeleton joints, in 3D space, Technologies 2019, 7, 56; doi:10.3390/technologies7030056 www.mdpi.com/journal/technologies Technologies 2019, 7, 56 2 of 18 which are then properly analyzed to extract dance kinematics [9]. The preservation of folk dances can be facilitated by modern Information and Communication Technologies by leveraging recent developments in a variety of areas, such as storage, image and video processing, machine learning, cloud computing, crowdsourcing, and automatic semantic annotation, to name a few [10]. Nevertheless, the digitization and the modelling of the information remains the most valuable task. Due to the tremendous growth of the motion-capturing systems, depth cameras are a popular solution employed in many cases, because of their reliability, cost-effectiveness, and usability and despite their limited range. Kinect is one of the most recognizable sensors in this category and in the choreography context can be used for recording sequences of points in 3D space for body joints at certain moments in time. Several recent research papers in the literature make use of such sensors for dance analysis, for example educational dance applications using sensors and gaming technologies [11], trajectory interpretation [12], advanced skeletal joints tracking [13], action or activity recognition [14–20], key pose identification [21] and key pose analysis [22]. Apart from Kinect, another popular alternative motion capture system is VICON which is significantly more sophisticated and accurate [9,23,24]. In [25], a comparison between abilities of the Kinect and VICON for gait analysis is introduced in the orthopedic and neurologic field. In [26], the authors focus on the precision of the Kinect and the VICON motion-capturing systems creating an application for rehabilitation treatments. In [27], the authors propose that the Kinect was able to accurately measure timing of clinically relevant movements in people with Parkinson disease. Contrary to the linear regression-based approaches that have been carried out in the bio-medical field [25–27] regarding the similarities/dissimilarities and the precision of the adopted motion-capturing system, in this work we follow a Dynamic Time Warping (DTW) approach in the kinesiology field. Moreover, the aforementioned approaches pertain to simple movement sequences, i.e., knee flexion and extension, hip flexion and extension instead of our proposed choreographic dataset which includes more complex movements that combine several joints variations (see Table1). In [28], the authors introduce a motion classification framework using DTW. The aforementioned work uses DTW algorithm to classify motion sequences using the minimum set of bones (7 body joints). On contrary, our proposed framework uses 25 body joints analyzing the motion sequences using the DTW and Move-Split-Merge algorithms, respectively. In [29], the authors propose an algorithm for 3D motion recognition which allows extensions of DTW with multiple sensors (view-point-weighted, fully weighted and motion-weighted) and can be employed in a variety of settings. DTW algorithm has also adopted to extract the kinesiology details from video sequences. In [30], the authors propose a video human motion recognition approach, which uses DTW to match motion projections in non-linear manifold space. In [31], the authors present a technique for motion pattern and action recognition, which employs DTW to match motion projections in Isomap non-linear manifold space. Technologies 2019, 7, 56 3 of 18 Table 1. Greek folklore dances and the main choreographic steps. Folklore Dance Dance Description Main Choreographic Steps Enteka (11) A popular dance in Western Macedonia (Kozani, (1) Initial Posture (IP); Kastoria, Greven, Florina, etc.). It is performed freely (2) Right Leg Up (RLU); as a street carnival dance, but also around the carnival (3) Dancer’s Right Turn fires. The dancers’ hands are free to move or can be (DRT); (4) Initial Posture placed at the waist. (IP) (5) Dancer’s Left Turn (DLT). Kalamatianos A very popular Greek traditional dance, also known (1) Initial Posture (IP); at an international level. It is a circle dance, which is (2) Cross Legs (CL); (3) Cross performed by dancers holding hands and moving in a Legs (CL); (4) Cross Legs counterclockwise circular manner. (CL); (5) Cross Legs (CL); (6) Initial Posture (IP); (7) Cross Legs Backwards (CLB). Makedonikos A circle dance (known mainly in Western and Central (1) Initial Posture (IP); Macedonia) which follows a 7/8 musical beat. It (2) Left Leg Back (LLB); is similar to Kalamatianos, since it also includes 12 (3) Cross Legs (CL); (4) Cross distinct steps, but it is performed in a more joyful Legs (CL); (5) Cross Legs fashion. (CL); (6) Initial Posture (IP); (7) Right Leg Back (RLB) Syrtos (2 beats) The Syrtos (2 beats) dance is structured in a quick (1) Initial Posture (IP); 2-beat rhythm. It is mainly popular in Epirus and it is (2) Cross Legs Backwards danced in a circular setting, both by men and women. (CLB); (3) Initial Posture (IP); (4) Left Leg Front (LLF); (5) Initial Posture (IP); (6) Left Leg Front (LLF) Syrtos (3 beats) Syrtos is structured in a slower 3-beat rhythm. It can (1) Initial Posture (IP); be danced both at a line and at a circular setting, both (2) Cross Leg (CL); (3) Initial by men and women, holding hands and facing right. Posture (IP); (4) Left Leg Up It is very popular throughout Greece and Cyprus. (LLU); (5) Initial Posture (IP); (6) Right Leg Up (RLU) Trehatos Trehatos is a dance stemming from the village (1) Initial Posture (IP); (Running) Neochorouda in Thessaloniki. Its kinetic theme (2) Cross Legs (CL); (3) Cross includes three different patterns: one resembles the Legs (CL); (4) Cross Legs Syrtos (3 beats) pattern, the second is characterized by (CL); (5) Initial Posture intense motor activity and the other one connects the (IP); (6) Left Leg Up (LLU); aforementioned two. It is danced in a circular setting. (7) Right Leg Up (RLU); (8) Left Leg Up (LLU); (9) Cross Legs Backwards (CLB) Our proposed framework focuses on the similarity assessment of folkloric dances, using data from heterogeneous sources; i.e.,