WEARABLE COMPUTING ARCHITECTURE OVER DISTRIBUTED DEEP LEARNING HIERARCHY: FALL DETECTION STUDY by XIAOYE QIAN Submitted in partial fulfillment of the requirements For the degree of Master of Science Department of Electrical Engineering and Computer Science CASE WESTERN RESERVE UNIVERSITY August, 2019 CASE WESTERN RESERVE UNIVERSITY SCHOOL OF GRADUATE STUDIES We hereby approve the thesis/dissertation of Xiaoye Qian candidate for the degree of Master of Science *. Committee Chair Ming-Chun Huang Committee Member Pan Li Committee Member An Wang Date of Defense May ퟑ풓풅, 2019 *We also certify that written approval has been obtained for any proprietary material contained therein. iii Table of Contents List of Tables iv List of Figuresv Acknowledgements vii Abstract viii Chapter 1. Introduction1 Chapter 2. Literature Review5 Chapter 3. System Overview8 Data Acquisition and Processing9 Distributed Hierarchical Deep Learning 12 Chapter 4. Experiment and Results 18 Experimental Setup 18 Result and Analysis 21 Chapter 5. Discussion 29 Why to use resources on the cloud server? 29 What are the requirements about the incoming data during the training process? 31 Why to use paired smart insole rather than only one smart insole? 31 Chapter 6. Conclusion 34 Chapter 7. Suggested Future Research 35 References 36 iv List of Tables 4.1 The main specs of the mobile device’s hardware. 18 4.2 The comparison experimental results of using smart insoles and wearable sensors placed on SOIs for fall detection,W-waist, H-hand, T-thigh, I-smart insole, with k-nearest neighbor (knn), support vector machines (svm), decision tree (dt), discriminant (dc),Multi-Layer Perceptron (mlp) and convolutional neural network (cnn). 22 4.3 The best results through the six algorithm studies. The combination represents the combination of smart Insole, smartwatch and smartphone. 23 4.4 Performance of two different neural networks CNN and MLP under two different parameter exchange protocols. 24 4.5 Running time and Running Memory usage. 27 5.1 The confusion matrix using CNN with only one smart insole as inputs. CM-confusion matrix, Fa-Falling, SD-Sit Down, Wa-Walking, BP-Bend to Pick up, St-Standing, Be-Bending, Si-Sitting, Ly-Lying, Sq-Squatting and SP-Squat to pick up 32 5.2 The confusion matrix using CNN with two smart insole as inputs. 33 v List of Figures 3.1 This figure shows the fall detection system flowchart. The sensor data collected from the smartwatch, smartphone and smart insoles. All the data are sent to the smartphone simultaneously. The smartphone will combine all three sensor data and perform deep learning through the system. During the training process, each smartphone transmits the intermediate output and label to the cloud server which will calculate corresponding gradient-outputs L(w,b;x, y) and release it back to rn j the smartphone. After finishing the training process, the smartphone will deliver the accumulated gradients to the cloud server, and the cloud server will update the consensus model according to these gradients. The updated weights will distribute to one or more smartphones which send the requests. 10 3.2 Overview of DHNN architecture. (a) is the traditional DNN architecture. (b) introduces HNN architecture including a single smartphone and the cloud server. And (c) illustrates DHNN architecture in which the system further extends to multiple mobile deices. 13 4.1 (i), (ii), (iii), (iv), (v) representing the changes of acceleration, angular velocity, and orientation on smartwatch and smartphone, as well as the moving trend of pressure center of the smart insoles when presenting five different kinds of fall movements. Plot (vi), (vii), (viii), (ix), (x) showing the changes when presenting five different ADLs. 21 vi 4.2 The distributed hierarchical deep learning architecture used in the system evaluation. The blocks in orange and blue represent layers working on the mobile devices and cloud server respectively. After the training process, the smartphones will upload the updates to the cloud server instantly. 25 4.3 The Loss and Accuracy in the iteration process. 26 4.4 The reconstruction rate of MLP and CNN in DHNN system with multiple local smartphones under two different parameter exchange protocols, Round Robin and Asynchronous. 28 5.1 The running time and memory usage with a different number of layers built on smartphones. The percentage of parameters of the neural network implemented on smartphones is also shown in the parentheses. 30 vii Acknowledgements This project would not have been possible without the support of many people. I would like to thanks to my advisor, Dr. Ming-Chun Huang of the department of electri- cal engineering and computer science at Case Western Reserve University. He gave me detailed instruction through the graduate student life. I also thank my lab mates and friends. Many thanks to my collaborator, Haotian Jiang who provided the android deep learning platform and Diliang Chen who design the smart insole. Also, this work made the use of the Case Western Reserve University high-performance computing resource. Finally, I show my huge gratitude to my parents and families. they offer me continu- ous supports and encourages and help me to complete the graduate study. viii Abstract Wearable Computing Architecture over Distributed Deep Learning Hierarchy: Fall Detection Study Abstract by XIAOYE QIAN With the development of technologies, the increasing number of mobile devices are used all around the world. Wearable sensors can provide quantitative assessments for fall-based movements. Detecting falls from multiple intelligent wearable sensors have aroused wide attention through academia and industry, because falls are the common incidents among human beings and lead to some serious consequences. Wearable sen- sors can provide quantitative assessments for human movements. Automatic fall detec- tion systems with the wearable sensors are becoming popular in recent years. In this manuscript, a novel data acquisition method is proposed in the fall detection system through the wearable gait lab including a smartwatch, a smartphone, and two smart in- soles. Deep learning has shown great potential for performing automatic fall detection. The proposed system applying a distributed hierarchical neural network (DHNN) archi- tecture over a cloud server and mobile devices based on machine learning algorithms. The system enables multiple mobile devices to train a shared consensus model collabo- ratively and takes advantage of the abundant computational resources on a cloud server to minimize the limitation of the computational and storage resource on mobile devices. ix The patients’ privacy is protected as well. Both quantitative and qualitative analysis are implemented through well-designed experiments. 1 1 Introduction Falls are the most common incident among human beings. It poses a global health problem. In the United States, more than 1.6 million adults receive treatment due to the fall-related accidents every year1, and the financial costs associated with fall are ris- ing in these years2. Approximately one-third of the aged population fall at least once a year, and the similar reports are also generated from other countries, such as Spain and Colombia3. Fall is considered as the most common incidents among human beings and leads to dangerous situations for the seniors4. With increasing age, the physical changes make people more prone to falls, and the fall injuries are exacerbated. It is the second most important cause of mortality. Specifically, 37.3 million falls requiring medical at- tention occur and around 646,000 people die every year as the result of falls5. Falls event leads to significant injuries including skin abrasions, upper limb and hip fractures, brain injuries and general connective tissue lesions5,6. Falls not only seriously threaten the health, but also cause the psychological problem like lowering the self-confidence and being afraid of independent life, which further weakens the quality of daily life7. In the past few decades, falls detection has attracted more attention from the public. Most of the time, the fallers might lose consciousness and are unable to call for help. Therefore, many automatic fall detection technologies have proposed in recent years8. Introduction 2 It is generally acknowledged that the Internet of Things (IoT) has to be widely used in healthcare applications all over the world. IoT provides one kind of interaction of the subjects, sensors, and computing devices. In many studies, the vision, sound, radar, and infrared sensors perform well on automatic falls detection. But, these ambient sensor- based technologies have the problem of privacy and make the seniors face many con- straints, for example, living in a restricted zone9–11. In the last decades, the development of the wearable sensor-device provides new chances for detecting fall-related accidents. The wearable sensor-based fall detection systems eliminate the space limitation com- pared with the systems based on the ambient technologies9. Due to the wearable ac- celerometers have the characters of small size and low price, many wearable fall detec- tion systems are designed based on such sensors and place them on different positions on the subject of interest (SOI). However, in these ways, SOIs have to wear many sen- sors during daily activities, which make them inconvenient. On the other hands, It is forgettable for the SOIs to wear complicated wearable sensors, especially for seniors. Nowadays, smart insoles based on the wearable