UCLA UCLA Electronic Theses and Dissertations Title A Framework for Pervasive Context Awareness Permalink https://escholarship.org/uc/item/7bj942gr Author Shen, Chenguang Publication Date 2016 Peer reviewed|Thesis/dissertation eScholarship.org Powered by the California Digital Library University of California University of California Los Angeles A Framework for Pervasive Context Awareness A dissertation submitted in partial satisfaction of the requirements for the degree Doctor of Philosophy in Computer Science by Chenguang Shen 2016 c Copyright by Chenguang Shen 2016 Abstract of the Dissertation A Framework for Pervasive Context Awareness by Chenguang Shen Doctor of Philosophy in Computer Science University of California, Los Angeles, 2016 Professor Mani B. Srivastava, Chair The proliferation of pervasive computing devices with unprecedented sensing, communica- tion, and computation capabilities has enabled continuous and ubiquitous monitoring of users and their surrounding environments. While smartphones have evolved from only communi- cation devices into powerful personal computing platforms, connected devices such as smart- watches, cameras, motions sensors, thermostats, and energy meters, collectively dubbed as the Internet of Things, are also rapidly permeating our living spaces. The continuous stream of richly annotated, real-time data made available by tapping into the spectrum of sensors available on these devices has led to the emergence of a sprawling ecosystem of context-aware apps. These apps use sensors such as GPS, microphone, accelerometer, and gyroscope to make diverse inferences about user activities and contexts. Typical context-aware apps employ a suite of machine learning algorithms to extract semantically meaningful inferences from sensor data. However, today's connected devices and mobile operating systems are not designed to support sensing and inference workloads. Compared with the rich network stack on pervasive devices, where one can leverage protocols and abstractions in different layers, context inference apps are composed in a monolithic fashion. That is, developers have to implement all data collection and inference logic using the raw data. As a result, the development and execution of context inference apps today are accompanied by myriad difficulties. First, the limited sensor coverage from running inferences on a single device and the need of hand-picking high-level features have made it challenging ii to achieve high inference accuracy in building the classification model of apps. Second, the continuous execution of complex algorithms on the main app processor of smartphones and the use of energy-hungry sensors have resulted in high energy consumption of inference apps, especially when they are executed on battery-powered pervasive devices. Finally, the monolithic development practice limits the adoption of inference apps onto heterogeneous devices, and requires enormous amount of efforts from app developers in composing the apps. In this dissertation we make three research contributions towards building a framework for pervasive context awareness: • We first showcase that context-aware inferences can benefit from heterogeneous con- nected devices such as smartwatches. We develop an example app to autonomously in- fer workout exercises of users from only sensors on a commercial smartwatch, achieving 90% classification accuracy for both cardio and weightlifting exercises while extending the watch battery life by up to 19 hours compared with prior approaches. • Having identified the problems from our example app, we then perform three optimiza- tions of context inference apps. We have achieved (1) comparable inference accuracy as traditional models and acceptable latency using deep learning without hand-picking features; (2) up to 30× speed-up of deep learning tasks using mobile GPUs and up to 60% energy saving of off-loading inference tasks from the CPU to the DSP; and (3) up to 37% accuracy improvement and up to 67% less energy consumption for context- aware apps from watch-phone coordinations. • Finally, we close the loop by proposing the design and implementation of a program- ming framework for context inference apps, with a set of programming abstractions and an associated runtime. The framework helps reduce development tasks by up to 4:5× and source lines of code by up to 12×. It also tackles runtime challenges and achieves 3× better inference accuracy by handling environmental dynamics. With this work we have also created an open-source toolkit for developing and executing context inference apps using heterogeneous pervasive devices. iii The dissertation of Chenguang Shen is approved. Tyson Condie William J. Kaiser Jens Palsberg Mani B. Srivastava, Committee Chair University of California, Los Angeles 2016 iv Table of Contents 1 Introduction :::::::::::::::::::::::::::::::::::::: 1 1.1 Contribution....................................5 1.1.1 Motivating Example: Leveraging Smartwatches for Context Inferences5 1.1.2 Accuracy and Energy Optimization of Context Inferences.......6 1.1.3 Closing the Loop: A Programming Framework for Context Inferences6 1.2 Related Work...................................7 1.2.1 Workout Tracking.............................8 1.2.2 Deep Learning on Mobile and Wearable Sensor Data.........9 1.2.3 Leveraging Heterogeneous Processors in Mobile SoC.......... 10 1.2.4 Context Inferences on Smartwatches.................. 11 1.2.5 Programming Framework for Context Inferences............ 12 1.3 Organization................................... 13 2 Motivating Example: Context Inferences on Smartwatches :::::::: 15 2.1 Motivation and Contribution........................... 15 2.2 Challenges and Design Choices......................... 18 2.2.1 C1: Single-device Sensing........................ 18 2.2.2 C2: Automatic Segmentation...................... 18 2.2.3 C3: Weightlifting Exercise Tracking................... 19 2.2.4 C4: Efficient Resource Usage....................... 19 2.3 System Architecture............................... 20 2.3.1 Overview................................. 20 2.3.2 High-level Activity Classification.................... 22 v 2.3.3 Weightlifting Classification........................ 24 2.3.4 Context-aware Optimization....................... 32 2.4 Implementation.................................. 34 2.5 Evaluation..................................... 35 2.5.1 MiLift Tracking Accuracy........................ 36 2.5.2 Energy Efficiency of MiLift Models................... 42 2.5.3 User Task and Battery Life Analysis.................. 42 2.6 Discussion..................................... 45 2.7 Summary..................................... 46 3 Accuracy and Energy Optimization of Context Inferences ::::::::: 47 3.1 Overview and Challenges............................. 47 3.1.1 Inference Accuracy............................ 47 3.1.2 Energy Consumption........................... 49 3.1.3 Design Choices.............................. 50 3.2 Applying Deep Learning for Mobile Context Inferences............ 51 3.2.1 Overview................................. 51 3.2.2 Task: High-level Activity Classification................. 52 3.2.3 Training Workflow............................ 53 3.2.4 Evaluation................................. 54 3.3 Exploiting Co-Processors in Mobile SoCs.................... 55 3.3.1 Background: Processor Heterogeneity.................. 55 3.3.2 Using Mobile GPUs for Deep Learning................. 56 3.3.3 Leveraging DSPs for Classification................... 58 3.3.4 Learning on DSPs............................. 66 vi 3.4 Wearable-Mobile Coordination.......................... 74 3.4.1 Background: Smartwatch Basic Profiles................. 75 3.4.2 Design Goal................................ 77 3.4.3 Implementation.............................. 80 3.4.4 Evaluation................................. 81 3.5 Summary..................................... 91 4 Putting It Together: A Programming Framework for Context Inferences 92 4.1 Design Challenge and Contribution....................... 92 4.2 Beam Overview.................................. 95 4.2.1 Example Apps.............................. 95 4.2.2 Beam Abstractions............................ 96 4.3 Beam Runtime.................................. 100 4.3.1 Device Selection.............................. 100 4.3.2 Inference Partitioning for Efficiency................... 101 4.3.3 Disconnection Tolerance......................... 103 4.4 Implementation.................................. 104 4.4.1 Sample apps................................ 104 4.4.2 APIs.................................... 106 4.5 Evaluation..................................... 108 4.5.1 Development Approaches......................... 109 4.5.2 Evaluation of Inference Abstraction................... 110 4.5.3 Device Selection.............................. 115 4.5.4 Efficient Resource Usage......................... 116 4.5.5 Handling Disconnections......................... 118 vii 4.6 Discussion..................................... 119 4.7 Summary..................................... 120 5 Context Awareness Toolkit :::::::::::::::::::::::::::: 122 5.1 Data Collector................................... 122 5.1.1 Design................................... 123 5.1.2 Implementation.............................. 124 5.2 Inference Composer................................ 125 5.2.1 Design................................... 126 5.2.2 Implementation.............................. 127 5.3 Inference Executor................................ 129 5.3.1 Design..................................