DOCSLIB.ORG

Voice Enabling Mobile Applications with Uivoice

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Voice Enabling Mobile Applications with Uivoice Voice enabling mobile applications with UIVoice Ahmad Bisher Tarakji Jian Xu∗ Samsung Research America Stony Brook University [email protected] [email protected] Juan A. Colmenares† Iqbal Mohomed Samsung Research America Samsung Research America [email protected] [email protected] ABSTRACT The key mechanism we introduce to tie voice agents with mo- Improvements in cloud-based speech recognition have led to an bile applications is User Interface Automation. That is, we enable 1 explosion in voice assistants, as bespoke devices in the home, cars, developers to automate UI interactions that a human user would wearables or on smart phones. In this paper, we present UIVoice, perform on a mobile app, and connect these to voice interactions through which we enable voice assistants (that heavily utilize the that an end user would engage with on an agent. This combination cloud) to dynamically interact with mobile applications running in lets us “voice enable” applications that were never designed for the edge. We present a framework that can be used by third party interface with voice agents. A novel facet of our approach is the developers to easily create Voice User Interfaces (VUIs) on top of ex- ability to have a dialogue with the user in the face of ambiguities. isting applications. We demonstrate the feasibility of our approach There are four contributions in this paper: (i) we present the through a prototype based on Android and Amazon Alexa, describe UIVoice system for creating Voice User Interfaces (VUIs) on top how we added voice to several popular applications and provide of existing voice agents and mobile applications, (ii) we describe a an initial performance evaluation. We also highlight research chal- framework that simplies the task of creating VUIs, (iii) we describe lenges that are relevant to the edge computing community. our prototype system, discuss sample VUIs for several popular mobile apps and present a detailed evaluation, and nally, (iv) we CCS CONCEPTS highlight interesting research problems in edge computing that emanate from our work. • Human-centered computing → Ubiquitous and mobile com- In this paper, we start by giving a background on voice assistants puting systems and tools; Interactive systems and tools; and how they currently interface with third-party applications (§2). In §3, we introduce our UIVoice system prototype that works with 1 INTRODUCTION Amazon’s Alexa family of voice agents and the Android OS. We Recently there has been great excitement on the possibilities oered use our prototype to create VUIs for several popular mobile apps, by edge computing to support applications in saving bandwidth, which are described and evaluated in §4. In §5, we discuss various providing low latency experiences to users, preserving user privacy, important issues that emerge from our research. We also provide etc. In this paper, we describe a novel edge application — extend- related work on automation techniques developed in industry and ing the capabilities of a voice assistant by performing actions on academia in §6, and the paper concludes in §7. a user’s personal mobile device. Modern voice assistants perform most automatic speech recognition (ASR) and natural language un- 2 BACKGROUND ON VOICE ASSISTANTS derstanding (NLU) functions in the cloud. On the other hand, mobile In recent years, voice-based interactions with computers have been devices contain signicant personal state of users, that which spans embodied in the persona of voice assistants or agents, which can across applications. By giving voice assistants access to the applica- be bespoke devices in the home, integrated into cars and wearables tions and state within mobile devices, we believe that an immense or on smartphones. At present, there is no standard software ar- degree of personalization can be achieved. However, this leads to chitecture for voice assistants. However, there are two conceptual interesting architecture and latency considerations that will be of operations that occur in processing user utterances. First, the user’s interest to the edge computing community. utterance goes through an automated speech recognition (ASR) module. The key output of this module is an attempted transcrip- ∗Work done during internship at Samsung Research America. †Work done at Samsung Research America. tion of what the user said - text albeit still with speech recognition errors. Next, the recognized text goes through a Natural Language Permission to make digital or hard copies of all or part of this work for personal or Understanding (NLU) module that attempts to extract the desired classroom use is granted without fee provided that copies are not made or distributed user intent. Advanced NLU/NLP may add more complex process- for prot or commercial advantage and that copies bear this notice and the full citation ing to the user’s voice utterance, such as emotion recognition and on the rst page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or speaker recognition. ASR and NLU systems that allow programma- republish, to post on servers or to redistribute to lists, requires prior specic permission bility, typically require the third-party developer to specify a set and/or a fee. Request permissions from [email protected]. of user utterances, and how to extract commands (subsequently EdgeSys’18, June 10–15, 2018, Munich, Germany © 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-5837-8/18/06...$15.00 1This can be any developer or even a power user - not necessarily the one who created https://doi.org/10.1145/3213344.3213353 the specic application(s) being automated. 49 EdgeSys’18, June 10–15, 2018, Munich, Germany A.B. Tarakji et al. referred to as Intents) and parameters (subsequently referred to as Slots) from the utterance. For instance “Call my Mom” would initiate a “call phone number” Intent with the Slot “Mom” of type “Contact” from which the assistant should extract the number to call. Of course, there may be multiple such applications active at any given point in time. Ultimately, the job of the speech processing system is to take the user utterance, extract the intent of the user request, and route the set of action and parameters to the appro- priate VUI application. One important thing to note about existing Figure 1: System design. voice assistants from a systems perspective is that the majority of ASR and Intent classication occurs in the cloud (a notable excep- tion is wake word recognition). Some systems do provide a limited and mobile phone are connected to the home network. Since most recognition vocabulary on-device (e.g. functions to set alarms or of the ASR and NLU for Alexa occur in the cloud, this results in make a call). This is a fundamental problem because the size of communication from the in-home device to the Alexa backend in the recognition vocabulary has an impact on the performance of the cloud. Logic in that tier connects with our UIVoice backend the ASR system (and its resource requirements). The prevalence of that is hosted in the cloud. Our UIVoice backend also maintains a cloud-based ASR and NLU leads to latency and other challenges communication channel to the mobile device running the UIVoice that are relevant to the edge computing community. agent. This setup introduces various wide-area latencies that we A key challenge with existing SDKs from major commercial ven- discuss further during our evaluation. dors is that they require mobile application developers to modify The results of the rst step are two-fold: 1) code and congu- their code to support VUIs [2]. Given the large number of applica- ration that needs to be uploaded to the voice agent system, and tions on major app stores, this is a signicant barrier to the use of 2) instructions that are made ready to be sent to an agent on the voice assistants on mobile phones. The approach proposed in this mobile device. In the execution step, the UIVoice agent must be paper allows unmodied applications to be voice enabled. running on the user’s phone, to enable remote interaction with Creators of VUI-based applications have always had to design the applications. In our implementation, we also have a backend good voice interactions regardless of the mechanism used to im- component that runs on an external server and is responsible for plement them. Designers utilize good-practices such as VUI pat- mediating between the user’s phone and the voice agent. In this terns [12, 13] and Grice’s maxims of conversation [10] as a guideline section we describe all of these components. to closer approximate human expectations. However, the task of good dialog design is more an art than exact science [3, 9]. This is 3.2 VUI Creation Step an important matter but not tackled by our work. In our system, In this step, which occurs at development time, a VUI developer the creator of the voice-enabled application uses the primitives of (this could be an end user who knows how to program or some the voice agent to dene an eective dialog with the end-user. third-party developer) species the voice interactions they wish to support and the mapping to an existing mobile application. These 3 UIVOICE are both done by dening a Interaction Script. Conceptually, an In this section we present the UIVoice system overview and outline interaction script has both user-facing actions (that execute in the its two main steps. context of the voice agent) and on-device actions. On the Voice agent side, actions are “listen” to the user (to capture the command 3.1 System Overview the user wishes to execute for example) and “speak” to the user to “ask” them for input or “tell” them results.
Load more

Recommended publications
  • Gmail Integration Technical Reference Since Its Initial Release
    PureConnect® 2020 R1 Generated: 18-February-2020 Gmail Integration Content last updated: 20-June-2019 See Change Log for summary of Technical Reference changes. Abstract CIC offers support for Gmail. This document describes how to configure IMAP and SMTP to integrate with Gmail in CIC using OAuth protocol authentication. For the latest version of this document, see the PureConnect Documentation Library at: http://help.genesys.com/cic. For copyright and trademark information, see https://help.genesys.com/cic/desktop/copyright_and_trademark_information.htm. 1 Table of Contents Table of Contents 2 Introduction to the CIC Integration with Gmail 3 About Gmail 3 About CIC 3 Requirements 3 Concepts and terminology 3 Configuration Overview 4 Re-run IC Setup Assistant 4 Configure the Google Domain 5 Configure Mail in Interaction Administrator 5 Configure a Mailbox in Interaction Administrator 6 Test the Integration 6 Change Log 7 2 Introduction to the CIC Integration with Gmail Integration of CIC with Gmail provides outsourcing of email capabilities and using OAuth with the integration provides a safer environment. About Gmail Gmail, part of Google Apps for Work, is a web-based hosted email application offering message threading, message labels, message search, and spam filtering. About CIC Customer Interaction Center (CIC) is a suite of contact center applications that has interactive voice response and IP/PBX capabilities. CIC supports several interaction management client applications. Starting with CIC 2015 R3, Interaction Desktop replaces Interaction Client .NET Edition as the primary CIC client. Requirements Starting with CIC 2015 R2, the integration supports OAuth 2.0. Following are the requirements for the CIC Integration with Gmail: Gmail with OAuth 2.0 CIC 2015 R2 or later Note: OAuth is only available for customers using Google Apps for Work.
    [Show full text]
  • FIT to WORK: IMPROVING the SECURITY of MONITORED EMPLOYEES' HEALTH DATA Elizabeth A. Brown INTRODUCTION Imagine Coming to Work
    FIT TO WORK: IMPROVING THE SECURITY OF MONITORED EMPLOYEES' HEALTH DATA Elizabeth A. Brown1 INTRODUCTION Imagine coming to work one day and finding that your employer has given everyone in the company a wearable FitBit health monitor, free of charge. You pop the FitBit on, grateful for another bit of help in managing the health concerns that nag at you persistently but which never quite rise to the top of your priority list. At your next performance review, your supervisor expresses concern about your anxiety levels. Although your work output is slightly off, she notes, there has been a correlation in your lack of sleep and exercise, and she suspects you are depressed. You wonder how your employer might know these things, whether or not they are true, and then you remember the FitBit. Your supervisor then tells you that the promotion you had wanted is going to a colleague who is “better equipped to handle the demands of the job.” You interview for another job, and are asked to provide the password to the HealthDrive account that centralizes the fitness data all the apps on your iPhone collect about you during the day. Similar scenarios are playing out now in workplaces across the country, and will do so more frequently as the personal health sensor market and employee monitoring trends continue to grow. Employers are making key decisions based on employees’ biometric data, collected from specialized devices like a FitBit or the health-related apps installed on mobile phones. BP, for example, adjusts its employees’ health care premiums depending on how much physical activity their wearable FitBit devices monitor – devices that BP provides to thousands of employees, their spouses, and retirees for free.
    [Show full text]
  • Your Voice Assistant Is Mine: How to Abuse Speakers to Steal Information and Control Your Phone ∗ †
    Your Voice Assistant is Mine: How to Abuse Speakers to Steal Information and Control Your Phone ∗ y Wenrui Diao, Xiangyu Liu, Zhe Zhou, and Kehuan Zhang Department of Information Engineering The Chinese University of Hong Kong {dw013, lx012, zz113, khzhang}@ie.cuhk.edu.hk ABSTRACT General Terms Previous research about sensor based attacks on Android platform Security focused mainly on accessing or controlling over sensitive compo- nents, such as camera, microphone and GPS. These approaches Keywords obtain data from sensors directly and need corresponding sensor invoking permissions. Android Security; Speaker; Voice Assistant; Permission Bypass- This paper presents a novel approach (GVS-Attack) to launch ing; Zero Permission Attack permission bypassing attacks from a zero-permission Android application (VoicEmployer) through the phone speaker. The idea of 1. INTRODUCTION GVS-Attack is to utilize an Android system built-in voice assistant In recent years, smartphones are becoming more and more popu- module – Google Voice Search. With Android Intent mechanism, lar, among which Android OS pushed past 80% market share [32]. VoicEmployer can bring Google Voice Search to foreground, and One attraction of smartphones is that users can install applications then plays prepared audio files (like “call number 1234 5678”) in (apps for short) as their wishes conveniently. But this convenience the background. Google Voice Search can recognize this voice also brings serious problems of malicious application, which have command and perform corresponding operations. With ingenious been noticed by both academic and industry fields. According to design, our GVS-Attack can forge SMS/Email, access privacy Kaspersky’s annual security report [34], Android platform attracted information, transmit sensitive data and achieve remote control a whopping 98.05% of known malware in 2013.
    [Show full text]
  • “Speaktoit”- Voice Over Assistant Triadon
    Volume 6, Issue 5, May – 2021 International Journal of Innovative Science and Research Technology ISSN No:-2456-2165 “Speaktoit”- Voice over Assistant Triadon Priyanshu Kumar Pandey Md. Akhtar Mansuri Akarsh B. Tech B. Tech B. Tech Computing Science and Engineering Computing Science and Engineering Computing Science and Engineering Faridabad, Haryana Deoria, Uttar Pradesh Ghaziabad, Uttar Pradesh Abstract:- Voice assistants are software programs that This software application uses a normal voice or synchronizes human speech and responds to that language user interface to answer questions, give command. Voice assistants are boon for our generation recommendations, and perform activities by delivering as it has been made to make us ease at work, we can now requests from a set of web services. just order the assistants and get the desired result. It shows that the hidden voice commands that are Users can make their assistant do a variety of jobs, incomprehensible to people can control the VAs. A virtual like media playback, google, Wikipedia, and many other assistant is a voice over assistant that performs on voice to-do lists, by their verbal command. recognition method, using natural language, and speech synchronizing to provide an ease to users through phones The use of voice assistants is increasing heavily in and voice recognition applications. every sector. Every company, every developer is introducing their own version of Vas. Like google has its II. LITERATURE REVIEW own assistant “google home” Amazon has “Alexa” Apple has “Siri” etc. “TRIADON” voice assistant is one like An online ordering command method that uses the them but more secure as the privacy issues which has voice user interface in a group of objects.
    [Show full text]
  • Project Plan
    INTELLIGENT VOICE ASSISTANT Bachelor Thesis Spring 2012 School of Health and Society Department Computer Science Computer Software Development Intelligent Voice Assistant Writer Shen Hui Song Qunying Instructor Andreas Nilsson Examiner Christian Andersson INTELLIGENT VOICE ASSISTANT School of Health and Society Department Computer Science Kristianstad University SE-291 88 Kristianstad Sweden Author, Program and Year: Song Qunying, Bachelor in Computer Software Development 2012 Shen Hui, Bachelor in Computer Software Development 2012 Instructor: Andreas Nilsson, School of Health and Society, HKr Examination: This graduation work on 15 higher education credits is a part of the requirements for a Degree of Bachelor in Computer Software Development (as specified in the English translation) Title: Intelligent Voice Assistant Abstract: This project includes an implementation of an intelligent voice recognition assistant for Android where functionality on current existing applications on other platforms is compared. Until this day, there has not been any good alternative for Android, so this project aims to implement a voice assistant for the Android platform while describing the difficulties and challenges that lies in this task. Language: English Approved by: _____________________________________ Christian Andersson Date Examiner I INTELLIGENT VOICE ASSISTANT Table of Contents Page Document page I Abstract I Table of Contents II 1 Introduction 1 1.1 Context 1 1.2 Aim and Purpose 2 1.3 Method and Resources 3 1.4 Project Work Organization 7
    [Show full text]
  • Understanding and Mitigating Security Risks of Voice-Controlled Third-Party Functions on Virtual Personal Assistant Systems
    Dangerous Skills: Understanding and Mitigating Security Risks of Voice-Controlled Third-Party Functions on Virtual Personal Assistant Systems Nan Zhang∗, Xianghang Mi∗, Xuan Fengy∗, XiaoFeng Wang∗, Yuan Tianz and Feng Qian∗ ∗Indiana University, Bloomington Email: fnz3, xmi, xw7, [email protected] yBeijing Key Laboratory of IOT Information Security Technology, Institute of Information Engineering, CAS, China Email: [email protected] zUniversity of Virginia Email: [email protected] Abstract—Virtual personal assistants (VPA) (e.g., Amazon skills by Amazon and actions by Google1) to offer further Alexa and Google Assistant) today mostly rely on the voice helps to the end users, for example, order food, manage bank channel to communicate with their users, which however is accounts and text friends. In the past year, these ecosystems known to be vulnerable, lacking proper authentication (from the user to the VPA). A new authentication challenge, from the VPA are expanding at a breathtaking pace: Amazon claims that service to the user, has emerged with the rapid growth of the VPA already 25,000 skills have been uploaded to its skill market to ecosystem, which allows a third party to publish a function (called support its VPA (including the Alexa service running through skill) for the service and therefore can be exploited to spread Amazon Echo) [1] and Google also has more than one thousand malicious skills to a large audience during their interactions actions available on its market for its Google Home system with smart speakers like Amazon Echo and Google Home. In this paper, we report a study that concludes such remote, large- (powered by Google Assistant).
    [Show full text]
  • Android Galaxy Updated 4/7/2020 V 1.0
    Connect Through Tech Tech Coaching Quick Start Guide Using a Smartphone to Access Resources Google Calendar Android Galaxy updated 4/7/2020 v 1.0 ©2019 DOROT, Inc. For public use; please credit DOROT if possible Connect Through Tech Table of Contents Benefits 3 Prerequisites 4 Introduction 5 Creating an Appointment By Voice 7 By Hand 9 Other Appointment Options Recurring Appointments 20 Inviting Others 27 Reminders For All Appointments 34 For a Specific Appointment 40 Checking Your Calendar By Voice 44 By Hand 46 Changing Appointment 51 Canceling Appointment 56 Page 2 Connect Through Tech Benefits One of the reasons cell phones are so popular is because they are portable. This means all the things they are capable of doing, you can do “on the go.” You don’t need to pre-plan anything and you don’t need to be at home to take advantage of the help an app can provide. This guide covers the Google Calendar app, a free, pre- loaded app for Android Phones. Calendar is available for use with no set up and no fees to purchase or use. The Calendar app has many advantages over traditional paper calendars. Some functions can be done by voice, instead of by hand. Calendar appointments can be made to repeat daily, weekly, monthly or annually when you do the first entry – a big convenience for items that recur regularly. You can even ask Calendar to remind you before an appointment so you don’t forget. Calendar can also send invitations to other people to join you at particular events or appointments, complete with the location.
    [Show full text]
  • Devices, the Weak Link in Achieving an Open Internet
    Smartphones, tablets, voice assistants... DEVICES, THE WEAK LINK IN ACHIEVING AN OPEN INTERNET Report on their limitations and proposals for corrective measures French République February 2018 Devices, the weak link in achieving an open internet Content 1 Introduction ..................................................................................................................................... 5 2 End-user devices’ possible or probable evolution .......................................................................... 7 2.1 Different development models for the main internet access devices .................................... 7 2.1.1 Increasingly mobile internet access in France, and in Europe, controlled by two main players 7 2.1.2 In China, mobile internet access from the onset, with a larger selection of smartphones .................................................................................................................................. 12 2.2 Features that could prove decisive in users’ choice of an internet access device ................ 14 2.2.1 Artificial intelligence, an additional level of intelligence in devices .............................. 14 2.2.2 Voice assistance, a feature designed to simplify commands ........................................ 15 2.2.3 Mobile payment: an indispensable feature for smartphones? ..................................... 15 2.2.4 Virtual reality and augmented reality, mere goodies or future must-haves for devices? 17 2.2.5 Advent of thin client devices: giving the cloud a bigger role?
    [Show full text]
  • Designing Personal Assistant Software for Task Management Using Semantic Web Technologies and Knowledge Databases
    Designing Personal Assistant Software for Task Management using Semantic Web Technologies and Knowledge Databases Purushotham Botla Working Paper CISL# 2013-11 May 2013 Composite Information Systems Laboratory (CISL) Sloan School of Management, Room E62-422 Massachusetts Institute of Technology Cambridge, MA 02142 Designing Personal Assistant Software for Task Management using Semantic Web Technologies and Knowledge Databases By Purushotham Botla B.E., Electronics Mumbai University, 1995 SUBMITTED TO THE SYSTEM DESIGN AND MANAGEMENT PROGRAM IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE IN ENGINEERING AND MANAGEMENT AT THE MASSACHUSETTS INSTITUTE OF TECHNOLOGY JUNE 2013 ©2013 Purushotham Botla. All rights reserved. The author hereby grants to MIT permission to reproduce and to distribute publicly paper and electronic copies of this thesis document in whole or in part in any medium now known or hereafter created. Signature of Author: ________________________________________________________________________ System Design and Management May 20, 2013 Certified by: __________________________________________________________________________________ Stuart Madnick John Norris Maguire Professor of Information Technologies, MIT Sloan School of Management & Professor of Engineering Systems, MIT School of Engineering Thesis Supervisor Accepted by: _________________________________________________________________________________ Patrick Hale Director, System Design and Management Program This page left intentionally blank ii Designing Personal Assistant Software for Task Management using Semantic Web Technologies and Knowledge Databases By Purushotham Botla Submitted to the System Design and Management Program on May 20, 2013 in Partial Fulfillment of the Requirements for the Degree of Master of Science in Engineering and Management Abstract Adoption of social network sites and use of smart phones with number of sensors in them has digitized user’s activities in real-time.
    [Show full text]
  • Remote Connect 2016
    The Convenience of Remote Connect 2016 TOYOTA APP SMARTWATCH GOOGLE AMAZON Remote Connect As a Companion of the Smartphone ASSISTANT ALEXA Toyota Action Toyota Skill Toyota offers an incredible array of convenience and connectivity features. These features now includeGoogle Assistant and Amazon Alexa3 capability – as well as smartwatch integration – for 2018 and later models equipped with Remote Connect². KEY FOB WITH REMOTE FUNCTIONALITY Vehicles equipped with Remote Connect² have key fob13 compatibility for Remote Start1. Connected Services registration will be required to use the complete suite of Remote Connect services, which include Smartphone, Smartwatch, and smart home devices. Audio Plus vehicle key fob functionality is available for up to 3 years. Beyond 3 years requires a subscription. Applicable for select Model Year 2018 through 2020 Remote Connect capable vehicles. Select Model Year 2020 Remote Connect capable vehicles will have functionality for up to 10 years. Premium Audio vehicle key fob functionality is available for up to 10 years. Beyond 10 years requires a subscription. Applicable for select Model Year 2018 through 2020 Remote Connect capable vehicles. Using the key fob to remote start my Toyota: 1. Press the LOCK button on the remote. 2. Press the LOCK button a second time within 1 second. 3. Press the LOCK button again, this time holding it for 3 seconds. The engine will start. Note: Key Fob Remote Start will not function if Connected Services are waived. REMOTE CONNECT EQUIPPED VEHICLES BUILT BEFORE 11/12/18 Remote Connect equipped vehicles built before 11/12/18 were required to have an active Remote Connect trial or paid subscription for the key fob to perform remote start functionality.
    [Show full text]
  • Digital Forensic Analysis of Smart Watches
    TALLINN UNIVERSITY OF TECHNOLOGY School of Information Technologies Kehinde Omotola Adebayo (174449IVSB) DIGITAL FORENSIC ANALYSIS OF SMART WATCHES Bachelor’s Thesis Supervisor: Hayretdin Bahsi Research Professor Tallinn 2020 TALLINNA TEHNIKAÜLIKOOL Infotehnoloogia teaduskond Kehinde Omotola Adebayo (174449IVSB) NUTIKELLADE DIGITAALKRIMINALISTIKA Bachelor’s Thesis Juhendaja: Hayretdin Bahsi Research Professor Tallinn 2020 Author’s declaration of originality I hereby certify that I am the sole author of this thesis. All the used materials, references to the literature and the work of others have been referred to. This thesis has not been presented for examination anywhere else. Author: Kehinde Omotola Adebayo 30.04.2020 3 Abstract As wearable technology is becoming increasingly popular amongst consumers and projected to continue to increase in popularity they become probable significant source of digital evidence. One category of wearable technology is smart watches and they provide capabilities to receive instant messaging, SMS, email notifications, answering of calls, internet browsing, fitness tracking etc. which can be a great source of digital artefacts. The aim of this thesis is to analyze Samsung Gear S3 Frontier and Fitbit Versa Smartwatches, after which we present findings alongside the limitations encountered. Our result shows that we can recover significant artefacts from the Samsung Gear S3 Frontier, also more data can be recovered from Samsung Gear S3 Frontier than the accompanying mobile phone. We recovered significant data that can serve as digital evidence, we also provided a mapping that would enable investigators and forensic examiners work faster as they are shown where to look for information in the course of an investigation. We also presented the result of investigating Fitbit Versa significant artefacts like Heart rate, sleep, exercise and personal data like age, weight and height of the user of the device, this shows this device contains artefacts that might prove useful for forensic investigators and examiners.
    [Show full text]
  • Adattester: Secure Online Mobile Advertisement Attestation Using Trustzone
    AdAttester: Secure Online Mobile Advertisement Attestation Using TrustZone Wenhao Li, Haibo Li, Haibo Chen, Yubin Xia Institute of Parallel and Distributed Systems Shanghai Jiao Tong University ABSTRACT 1billiondollarsin2013duetothesefraudsandaroundonethird Mobile advertisement (ad for short) is a major financial pillar for of mobile ad clicks may constitute click-spam [18]. The most re- developers to provide free mobile apps. However, it is frequently cent research study [33] shows that, one of the largest click fraud thwarted by ad fraud, where rogue code tricks ad providers by forg- botnets, called ZeroAccess, induces advertising losses on the or- ing ad display or user clicks, or both. With the mobile ad market der of $100,000 per day. Ad frauds can typically be characterized growing drastically (e.g., from $8.76 billion in 2012 to $17.96 bil- into two types [26]: (1) Bot-driven frauds employ bot networks to lion in 2013), it is vitally important to provide a verifiable mobile initiate forged ad impressions and clicks; (2) Interaction frauds ma- ad framework to detect and prevent ad frauds. Unfortunately,this nipulate visual layouts of ads to trigger ad impressions and unaware is notoriously hard as mobile ads usually run in an execution envi- clicks from the end users. ronment with a huge TCB. Because of the urgent need to detecting mobile ad frauds, prior This paper proposes a verifiable mobile ad framework called approaches have made important first steps by using an offline- AdAttester, based on ARM’s TrustZone technology. AdAttester based approach [16, 26]. Specifically, they trigger the execution provides two novel security primitives, namely unforgeable clicks of mobile apps in a controlled environment to observe deviated and verifiable display.Thetwoprimitivesattestthatad-relatedop- behavior to detect ad frauds.
    [Show full text]