Human Computer Interaction System

HUMAN COMPUTER INTERACTION SYSTEM

FOR SELF-DRIVEN GOLF CART

A Project

Presented to the

Faculty of

California State Polytechnic University, Pomona

In Partial Fulfillment

Of the Requirements for the Degree

Master of Science

Computer Science

Hardeep Singh

2018

SIGNATURE PAGE

PROJECT: HUMAN COMPUTER INTERACTION SYSTEM FOR SELF-DRIVEN GOLF CART

AUTHOR: Hardeep Singh

DATE SUBMITTED: Spring 2018

Computer Science Department

Dr. Amar Raheja Project Committee Chair Professor of Computer Science

Dr. Daisy Tang Project Committee Member Professor of Computer Science

ABSTRACT

In this era of machine learning and sensors, goal of industry is to make faster computers capable of performing high end tasks without human interaction. Autonomous computers are being developed to perform human-like tasks, with hope to eliminate human errors.

Among the many tasks performed by humans every day, an absolutely necessary task is operating a machine for transportation. Autonomous vehicles are being built around the globe to provide better and safer transportation for passengers. In the attempt to utilize modern technologies and striving for safer environment, Cal Poly Pomona is developing a self-driving golf cart, to drive passengers/visitors around campus. However, removing human interaction completely from golf-cart pose a great risk. An autonomous machine can navigate through pre-programmed scenarios, but it will always lack the ability to make human decisions. Therefore, in this research, an on-board computer is built to provide passengers with human-computer interaction platform. On-board computer consists multiple human interaction techniques (verbally or physical) along with vehicles movement information during a transit. Human computer interface features active route selection, path planning and navigation updates on map, to provide passengers with updates during their commute. On-board computer also provides basic vehicle directing controls, allowing passengers to maintain control over autonomous machine actions, at all times.

These features along with Android sensory data and Google location updates, allow onboard computer to provide a better and safer transit for its passengers.

iii

TABLE OF CONTENTS

SIGNATURE PAGE ...... ii ABSTRACT ...... iii LIST OF FIGURES ...... vi CHAPTER 1 INTRODUCTION ...... 1 CHAPTER 2 LITERATURE SURVEY ...... 3 2.1 VOICE RECOGNITION ...... 3 2.1.1 USER TO ROBOT SPEECH MODEL STRUCTURE ...... 3 2.1.2 TYPE OF SPEECH RECOGNITION ...... 5 2.1.3 GOOGLE VOICE RECOGNITION...... 7 2.2 USER INTERFACE (MANUEL SELECTION) ...... 8 2.2.1 USER SELECTION OPTIONS ...... 8 2.2.2 USER NAVIGATION OPTIONS ...... 8 2.3 GOOGLE MAP ...... 9 CHAPTER 3 PROJECT GOAL ...... 11 CHAPTER 4 METHODOLOGY ...... 13 4.1 PROVIDED DATA/PREPROCESSING ...... 13 4.2 APPROACH ...... 13 4.2.1 DATA BINDINGS ...... 13 4.2.2 PARSE BUILDING DATA XML ...... 15 4.2.3 SPEECH TO TEXT CONVERTOR ...... 17 4.2.4 TEXT TO SPEECH CONVERTOR ...... 21 4.2.5 GOOGLE LOCATION UPDATES ...... 24 4.2.6 GOOGLE MAPS PATH PLANNING ...... 26 4.2.7 SERVER PATH PLANNING ...... 30 4.2.8 SIMULATOR ...... 32 CHAPTER 5 DATA FLOWS ...... 34 5.1 GOOGLE MAP PATH WITH ACTIVE LOCATION UPDATES ...... 34 5.2 GOOGLE MAP PATH WITH SIMULATED LOCATION UPDATES ...... 35 5.3 FILE PROVIDED PATH WITH ACTIVE LOCATION UPDATES ...... 36 5.4 FILE PROVIDED PATH WITH SIMULATED LOCATION UPDATES ...... 37 CHAPTER 6 RESULTS ...... 38 6.1 SELECTION/MAIN SCREEN ...... 38 6.1.1 OPTIONS ...... 39 6.2 NAVIGATION SCREEN ...... 48 6.2.1 TRANSIT ...... 48 CHAPTER 7 EVALUATION OF RESULTS ...... 56 7.1 APPROACH ...... 56 7.2 RESULTS ...... 58 CHAPTER 8 FUTURE WORKS ...... 59 8.1 Enhanced Speech Recognition ...... 59

REFERENCES ...... 60 APPENDIX ...... 62

LIST OF FIGURES

Figure 1. Speech to Robot Command ...... 4

Figure 2. Data flow with Google Maps path and active location updates ...... 34

Figure 3. Data flow with Google Maps path and simulated location updates ...... 35

Figure 4. Data flow with way-point file path and active location updates ...... 36

Figure 5. Data flow with way-point file path and simulated location updates ...... 37

Figure 6. Main screen of application...... 39

Figure 7. Search Bar usuage ...... 40

Figure 8. Default Google speech recognition dialog ...... 41

Figure 9. A confirmation dialog to confirm user selection by speech or touch ...... 42

Figure 10. Map view when building is selected ...... 42

Figure 11. Bottom panel when building is selected ...... 43

Figure 12. Map view when building is not selected ...... 43

Figure 13. Bottom panel when building is not selected ...... 43

Figure 14. Building information list view ...... 44

Figure 15. A confirmation dialog to confirm user selection by speech or touch...... 44

Figure 16. Map view when building is selected ...... 45

Figure 17. Bottom panel when building is selected ...... 45

Figure 18. Map view when building is not selected ...... 46

Figure 19. Bottom panel when building is not selected ...... 46

Figure 20. Bottom panel building selected (before start over action) ...... 46

Figure 21. Bottom panel with start over action ...... 47

Figure 22. Different transit options ...... 47

vi Figure 23. Launch panel ...... 48

Figure 24. Launch screen action buttons ...... 49

Figure 25. In-route panel ...... 49

Figure 26. In-route action button ...... 49

Figure 27. Destination arrived panel with action button ...... 50

Figure 28. Google map path and active location updates, launch screen...... 51

Figure 29. Google map path and active location updates, in-route screen ...... 51

Figure 30. Google map path and simulated location updates, launch screen ...... 52

Figure 31. Google map path and simulated location updates, in-route screen ...... 52

Figure 32. Way-point file path and active location updates, launch screen ...... 53

Figure 33. Way-point file path and active location updates, in-route screen ...... 53

Figure 34. Way-point file path and simulated location updates, launch screen ...... 54

Figure 35. Way-point file path and simulated location updates, in-route screen ...... 54

Figure 36. Destination reached panel ...... 55

vii CHAPTER 1

INTRODUCTION

In this era of machine learning and sensors, goal of the industry is to make faster computers capable of performing high end tasks without human interaction. New autonomous computers, also known as robots are being developed to perform human-like tasks, with hope to eliminate human errors. Among the many tasks performed by humans every day, a most dangerous and necessary task is driving. Weather its driving a car or a work vehicle, it is absolute essential as mean of transportation. As the era of robotics grow, demand for autonomous vehicle raises, with a hope to eliminate human error. However, removing human interaction entirely from driving pose a greater risk than allowing full human interaction. Autonomous machines can drive vehicles but without small human interaction, machine will always lack the ability to make human decisions, human conscious. (Churchill, Bowser, Preece 2013) For example: best route might not be safest route, and optimal speed might not be the most suitable speed for current passenger or driver. The problem of preventing the complete elimination of human conscious from autonomous vehicles can be solved by providing an onboard interface will allows human to alter machine decisions when necessary. A HCI (Human Computer Interaction) interface, embedded within an autonomous machine or vehicles, allows humans to oversee every task performed by the robot. Even though, autonomous vehicles are assigned to drive automatically, a human interaction will give user the flexibility to decide the routes, destination and speed for best and efficient results.

HCI requires easy and fast accessible interface, allowing prompt modification to machine computed tasks. HCI systems demands to inherit the most common way of human

1 interaction, speech. Speech serve as crucial factor in HCI’s, allowing user to verbally express their commands and concern to machine. However, given the diversity among humans, everyone speaks and express differently which occasionally makes interaction very difficult with machine. (Shrivastava, Singhal, Das, Nair 2013) To avoid speech recognition issue, a touch interface is also considered for HCI interface. Touch interface ensures the accuracy and safety of the users. HCI model require the need of navigation layout, to provide information as the vehicle progress through its tasks.

2 CHAPTER 2

LITERATURE SURVEY

HCI have rich theoretical history across a variety of domains: machine learning, artificial intelligence, sensor recognition, remote sensing, voice recognition, vehicle operation, machine operation, cognitive theories and interfaces.

2.1 VOICE RECOGNITION

From a high vantage point, there are many type of voice recognition systems, each excelling in specific feature. However, all the voice recognition relies on model-based approach using machine learning.

2.1.1 USER TO ROBOT SPEECH MODEL STRUCTURE

Speech recognition (SR) is a crucial part of this HCI providing the ability to interact with robot verbally. User will have the ability to speak destination (“Building 8”, “Building 10”, so on) and golf cart will calculate the path and execute the transport. However, taking a command from user and then conveying it to robot requires multiple steps. These steps require the use of network, speech server, database and tablet UI, to perform efficiently.

SR program has set architecture which ensure the best results with multiple speech dialogues.

Figure 1. Speech to Robot Command

In this architecture user side application on tablet capture the user input and transmit raw input to speech recognition server (SRS: Julius). This robot server is independent from the server that connect with user application. (Shrivastava, Singhal, Das, Nair 2013) Julius is an open source server tool which receives transmitted audio in small data packages. There are many open source speech recognition program and can be used to replace Julius, for example Google Speech Recognition. SRS can be used as library within user side application and it can decipher audio from audio file or microphone. Once SRS break down the audio, it transmits the recognized speech text to bridge application. Bridge application is considered mid-way break point between robot and user provided commands. The bridge application ensure robot is connected and working before any commands are sent over the

4 network. (Shrivastava, Singhal, Das, Nair 2013) In golf cart, this is crucial because if robot is not replying, an automatic stop commands can be executed to ensure the safety of the passenger. As bridge application receive an acknowledgment, commands are transmitted to bridge application. Bridge application holds the context-free grammar of the commands.

Therefore, these commands can be broken down into small context-free grammar to be linked with opcode within robot. Bridge application transmit each command, word by word to robot-relay server in opcode. Robot-relay forwards the opcode to robot and shut down the network till commands are being executed. (Shrivastava, Singhal, Das, Nair 2013)

During execution an acknowledgement is sent back to server to indicate progress. This progress will be used update the status on the map and to insure stop when pedestrians are detected as an obstacle. The entire speech recognition architecture is acknowledged based, which allows the flexibility to perform checks, on each part of program. Acknowledgments also help ensure STOP’s when no acknowledgment is detected by the server for certain period. This helps assume, robot require re-direction and have lost its way. SRS also provide the flexibility to run multiple commands together, without disrupting the original/main tasks. (Constantine 2001) This will give user the ability to control the robot entirely by speech, if user senses, a stop or turn is required.

2.1.2 TYPE OF SPEECH RECOGNITION

Speech Recognition Models are trained using multiple language patterns and dialects.

These models seek the word sequence W which is most likely to produce from acoustic evidence A (Young, Barnard 1987)

Equation 1. Speech Recognition Model Learning Equation

Command Based Voice Recognition

Use audibly provided commands to a computer system, this allows the activation of specific functions when specific commands are spoken. For example, “OPEN” or

“CLOSE”. CBVR is limited to few words and can be combined with identifiers that are added to dictionary using file.

Discrete Voice Recognition/Isolated Word Voice Recognition

This process of voice recognition demands a pause must be present between each word.

DVR is used in programs where essence of word starts, and word end is crucial. A blank sound or no sounds helps the recognition mark start and stop indices. An acute example of

DVR usage is word or phrase conversion.

Continuous Voice Recognition/Connected Words Voice Recognition

CVR lift the limitation of discrete voice recognition, and do not require a pause between words to recognize speech. CVR can determine the beginning and ending of any word within a phrase without any indication by the user, making it very complex to use and implement.

Spontaneous Speech Voice Recognition

SSPR is an improved version of continuous voice recognition system, it provides the ability to understand human non-word phrases for example ‘umm’ or ‘ahh’. This can be very useful in-house assistant system because system will automatically recognize the non-word

6 phrases and will not process them as an actual word.

Speech Dependent/Independent Voice Recognition

SDVR is designed for specific speaker, the models are trained around person’s speech style and patterns.

SIVR is opposite of SDVR, which allow it to host wide range of audience. However, SIVR is much more complex than SDVR, which require intensive processing power.

Natural Language Voice Recognition

This is an addition to continuous or spontaneous speech recognition because it can understand questions or commands based on delivery of the speaker. This is broadly used in voice recognition system in phone and super computers to achieve accuracy and efficiency. (Chris 2013)

2.1.3 GOOGLE VOICE RECOGNITION

Given there are many open source speech recognition software’s which can be customized for any specification purpose, google voice recognition is still most optimal choice. Unlike other SR software’s google voice recognition can be used as a library within any application. It allows the ability to make API calls to google serves and get results back in real-time. Once a speech is record via application, it is distributed to 6 different google servers where neural networks are used to find the best results. This eliminates the need to perform expensive processing using user device ensuring the up most accuracy and efficiency for each speech pattern. The results are returned in text form in real time,

7 cancelling any delay which might occur during on device processing. Google Speech recognition also support 80 different languages and multiple accents for each language, provide an edge to HCI system with diverse users. (Chris 2013)

2.2 USER INTERFACE (MANUEL SELECTION)

HCI is built on a speech recognition system but a touch-based GUI is always essential to any HCI. Speech recognition is performing admirably most of the time but given the uncertainty of language and accents in the world. There is always a possibility of misinterpretation in the commands. Given a scenario, where commands cannot be recognized verbally, a touch-based system is necessary, which allows the user to select and perform operations.

2.2.1 USER SELECTION OPTIONS

A graphical user interface will include the list of all the buildings and user preferences.

This will allow the user to select any destination by touch. Given Pixel C tablet is also touch-based, this feature can easily be implemented. User will be given the list of the all the destination, in an easy graphical interface, along with brief information about the destination. User will also be able to command start or stop of vehicle for emergency purposes. Destination selection using touch will remove ambiguities which voice recognition might provide during pre-processing the data.

2.2.2 USER NAVIGATION OPTIONS

Once user selects a destination, graphics interface will transition to map interface and fetch

8 the destination data from database. The algorithm run in the background will command the motors to move, accordingly. Along with selection of destination, user will also be provided to control the speed of the cart, for best possible transport. This feature will not be provided in speech recognition program because it can be hazard, if interpreted incorrectly. (David, Brown, Fan 2010) However, user will be given to the flexibility to increase speed, decrease speed or stop the cart by touch. This will also remove any uncertainties from speech recognition program and give user more control over the vehicle.

2.3 GOOGLE MAP

A web application service which allows users to map the world around them. Google Map is one of the biggest mapping service in the industry. It provides 360-degree camera angles, real-time traffic updates and it proficiently show the path from point A to point B.

However, collecting all this information for complete accuracy is impossible, which force google to rely on third party companies to provide vector data. Google maps have many unique features which can be helpful in implementation of HCI.

Street View

Provides detail view of the street with 360-degree camera flexibility. In HCI street view will allow user to check their surrounding once destination is reached. (Alexis 2012)

Location Service

Provide detail information of device current location and will be proven useful in updating

HCI interface. (Alexis 2012) Location service also help layout a path from point A and B

9 given a set of co-ordinates. This will allow the interface to draw polylines on the map to give visual representation of the path to user.

10 CHAPTER 3

PROJECT GOAL

The goal of this project is to design an android application which will provide a human computer interface for a self-driving golf cart:

• Develop interface software system to provide an open HMI display structure,

allowing easy access to current state of the machine.

• Develop interface software system to request the points of interest (POI) way-

points from the user(s).

(a) These way points will be building destination selected by the user at the

beginning of the transit.

• Develop interface software system to use moving mapping function during route

travel indicating the current vehicle position.

(a) Position will be indicated by a small golf-cart logo on the maps, giving user

easy perspective of gold-cart progress.

• Develop user interface software system to provide zoom in zoom out features on

the map, providing flexibility to look around the institute map.

• Design user interface software system to display the current route and reroute

computed by the system.

(a) Interface will allow the user to make selection and begin transit toward

selected destination. Along with route/reroute information, user interface will

provide POI (Point of Interest) information as the passengers’ travel.

• Develop user interface software system to support multiple modalities of user

requests, allowing user to make selection visually or aurally.

11 (a) To satisfy visual modalities, user software system will support multiple

display for user interaction, allowing user to utilize the golf-cart conveniently.

12 CHAPTER 4

METHODOLOGY

4.1 PROVIDED DATA/PREPROCESSING

To lay down an appropriate path for golf cart, path planning algorithms need to run on back-end and provide way points for golf cart to follow. These way points can be provided to robot in different metric but google maps will require co-ordinates to show route on HCI.

An initial delay between painting the path on maps and calculating the path after selection will also be expected. A pre-processing of building co-ordinates will also be required to avoid any delay in getting building information for the user. This will ensure building selection information is always available via speech or touch.

4.2 APPROACH

4.2.1 DATA BINDINGS

Data binding is part of android architecture to pass data between controller and view without explicitly setting data value to each view property. Data binding eliminates need the need for declare each android view with findViewById(R.id.view_name) and manually updating data with declared reference.

MVVM:

Data binding approach is not optimized till it’s used with MVVM (model view view- model) design pattern rather than traditional design patter MVC (model view controller).

MVVM introduce a new constant which always reflects the state of view. This allows user to bind a constant class with view. Upon updates, user update the class attributes with new values and view automatically reflect changes. Hence, it eliminates the need of

13 setting new values to view properties one by one.

In Android, view-models are very helpful because as user sends the application background, android operating system dispose the view to create space for active application. Upon reactivation of background application, operating system must redraw the view and resume application. However, with view-model approach, view-model is cached in memory and upon re-invocation view is rendered using the cached view-model class. This approach optimizes the operating system and memory utilities.

In this project, building data is binded using view-model and Building.java class has appropriate getter and setters which gets invoked automatically by android to reflect the right state of view according to the provided data.

public class Building extends BaseObservable implements Serializable { private String id; private String name; private String number; private String fullName; private String description; private String surroundings; private double latitude; private double longitude;

public Building() {}

public Building(String id, String name, String surroundings, double latitude, double longitude) { this.id = id; this.fullName = name; this.surroundings = surroundings; this.latitude = latitude; this.longitude = longitude; }

14 @Bindable public String getId() { return id; }

public void setId(String id) { this.id = id; notifyPropertyChanged(BR.id); }

@Bindable public String getFullName() { return fullName; }

public void setFullName(String fullName) { this.fullName = fullName; notifyPropertyChanged(BR.fullName); }

Note: Full Code (Appendix A)

4.2.2 PARSE BUILDING DATA XML

As per project requirements, building information will be provided in form of xml. XML contains building name, number, description, surrounding, latitude and longitude. This xml is placed in assets folder and parsed on start of application. This presented the need of XML parser and data feeder which can send all the collected data to the views.

Android provide XMLParserFactory which allows easy parsing of the xml

(Implementation). However, once xml is parsed, it needs to be sort for building info. All the data is stored in ArrayList of Building object. This arraylist is passed to the

RecyclerView which provide a list of buildings to user in a suitable view (Usage).

15 Implementation

public static ArrayList parseXML(InputStream xml) { ArrayList buildings = new ArrayList<>(); XmlPullParserFactory factory; XmlPullParser parser;

try { factory = XmlPullParserFactory.newInstance(); factory.setNamespaceAware(true); parser = factory.newPullParser(); parser.setInput(xml, null); int eventType = parser.getEventType(); Building building = null; String text = null; while (eventType != XmlPullParser.END_DOCUMENT) { String tagName = parser.getName(); switch (eventType) { case XmlPullParser.START_TAG: tagName = parser.getName(); if (tagName.equalsIgnoreCase(BUILDING)) { building = new Building(); } break; case XmlPullParser.TEXT: text = parser.getText(); break; case XmlPullParser.END_TAG: if (tagName.equalsIgnoreCase(BUILDING)) { buildings.add(building); } else if (tagName.equalsIgnoreCase(ID)) { building.setId(text); } else if (tagName.equalsIgnoreCase(NAME)) { building.setFullName(text); building.parseName(); } else if (tagName.equalsIgnoreCase(LATITUDE)) { building.setLatitude(Double.parseDouble(text)); } else if (tagName.equalsIgnoreCase(LONGITUDE)) { building.setLongitude(Double.parseDouble(text)); } else if (tagName.equalsIgnoreCase(DESCRIPTION)) { building.setSurroundings(text); } break; default: break; } eventType = parser.next();

Note: Full Code (Appendix J)

16 Usage

public void readBuildingsData() { AssetManager assetManager = getAssets(); InputStream inputStream = null; try { inputStream = assetManager.open(BUILDING_FILE); } catch (IOException e) { Log.e("Get XML Assets", e.getMessage()); } buildings = OnBoardUtil.parseXML(inputStream); }

/…In another place…/ recycleView.setLayoutManager(gridLayoutManager); buildingAdapter = new BuildingAdapter(buildings, this); recycleView.setAdapter(buildingAdapter);

Note: Full Code (Appendix K)

4.2.3 SPEECH TO TEXT CONVERTOR

Application allow users to provide verbal inputs. User can select building by saying a building number and confirm selection aurally. To handle speech input, a speech-to-text convertor was created to abstract out speech recognition functionalities. This class provide a way to listen to user input via default google dialog or with a custom dialog

(confirmation dialog). This class triggered only accessed by building selection screen

(Usage) and all speech recognition is handled by android default RecognizeListener

(Implementation). However, once speech is delivered a callback interface is required to deliver text result back to calling activity (Callback).

17 Implementation

/** * Listen without Dialog for specified SpeechRecognizer * * @param speechRecognizer */ public void startListening(final SpeechRecognizer speechRecognizer) { if (speechRecognizer != null) { activity.runOnUiThread(new Runnable() { @Override public void run() { speechRecognizer.startListening(getSpeechIntent()); } }); } }

/** * Stop Listening for specified SpeechRecognizer * @param speechRecognizer */ public void stopListening(final SpeechRecognizer speechRecognizer) { if (speechRecognizer != null) { activity.runOnUiThread(new Runnable() { @Override public void run() { speechRecognizer.stopListening(); } }); } }

Note: Full Code (Appendix F)

Usage

@Override public boolean onOptionsItemSelected(MenuItem item) { switch (item.getItemId()) { case R.id.action_search: return true; case R.id.action_mic: //Deliver message then listen for speech input by starting a speech dialog showProgressBar(); textToSpeechConverter.speakMessage("Speak Desired Building Number", SpeechDialogType.DIALOG); default: return super.onOptionsItemSelected(item); } }

Note: Full Code (Appendix K)

18 switch (speechDialogType) { case DIALOG:

speechToTextConverter.speechWithDialog(REQ_CODE_SPEECH_INPUT_W_DIALOG); break; case WITHOUT_DIALOG: speechToTextConverter.startListening(speechRecognizer); break; }

/** * Handle Speech Result with Google Dialog sent by onActivityResult * * @param speechResult */ public void cipherSpeechInput(String speechResult) { for (Building b : buildings) { if (b.getName().toLowerCase().contains(speechResult) || b.getFullName().toLowerCase().contains(speechResult)) { selectedBuilding = b; makeConfirmAlertDialog(this, "Did you say " + b.getName() + "?"); return; } else { showProgressBar(); textToSpeechConverter.speakMessage("I am sorry I didn't get that! Try again!", SpeechDialogType.DIALOG); }

Note: Full Code (Appendix K)

/** * Handle Without Dialog Speech Results for Alert Dialog * * @param results */ public void handleAlertDialogSpeechResults(ArrayList results) { textToSpeechConverter.finish(); if (results != null) { if (results.contains("yes") || results.contains("Yes") || results.contains("YES")) { updateSelectedBuildingInfo(); placeMarkerOnSelection(); alert.dismiss(); } else if (results.contains("no") || results.contains("No") || results.contains("NO")) { if (alert != null) { alert.dismiss(); } } else { showProgressBar(); textToSpeechConverter.speakMessage("I am sorry I didn't get that! Try again!", SpeechDialogType.WITHOUT_DIALOG); } }

Note: Full Code (Appendix K)

19 Callback

/** * Provide Results Callback from Speech To Text or Text to Speech * * @author Hardeep Singh ([email protected]) * December 28,2018 */ public interface ConversionDelegate {

void onSuccess(ArrayList result);

void onCompletion(SpeechDialogType speechDialogType);

void onErrorOccurred(String errorMessage); }

Note: Full Code (Appendix H)

/** * Handle Result for Custom Speech Recognition * Text To Speech Return Result on onCompletion() * Speech To Text Return Result on onSuccess() * * @param results */ @Override public void onSuccess(ArrayList results) { handleAlertDialogSpeechResults(results); }

@Override public void onCompletion(SpeechDialogType speechDialogType) { hideProgressBar(); switch (speechDialogType) { case DIALOG:

speechToTextConverter.speechWithDialog(REQ_CODE_SPEECH_INPUT_W_DIALOG); break; case WITHOUT_DIALOG: speechToTextConverter.startListening(speechRecognizer); break; } }

@Override public void onErrorOccurred(String errorMessage) { Log.e("SpeechRecognition: ", errorMessage); }

Note: Full Code (Appendix K)

20 4.2.4 TEXT TO SPEECH CONVERTOR

Application can also deliver message and ask questions, aurally. User can invoke speech listening dialog by selecting a mic on main screen, before a dialog is shown a message is delivered to user aurally to pronounce the building number. As user pronounce selection, a question is asked verbally to confirm their selection. To handle speech output, a text-to- speech convertor was created to abstract out speech output functionalities. This class provide a way deliver any message to user verbally. This class triggered only accessed by building selection screen (Usage) and all speech recognition is handled by android default RecognizeListener (Implementation). However, once speech is delivered a callback interface is required to provide success callback to calling activity (Callback).

21 Implementation

/** * Text to Speech Helper class * * @author Hardeep Singh ([email protected]) * March 28,2018 */ public class TextToSpeechConverter {

private ConversionDelegate conversionDelegate; private Activity activity; private TextToSpeech textToSpeech; private String utteranceId = this.hashCode() + "";

public TextToSpeechConverter(ConversionDelegate conversionDelegate) { this.conversionDelegate = conversionDelegate; this.activity = (Activity) conversionDelegate; }

/** * Speak Message and Attach Progress Callback * @param message * @param speechDialogType * @return */ public TextToSpeechConverter speakMessage(final String message, final SpeechDialogType speechDialogType) { textToSpeech = new TextToSpeech(activity, new TextToSpeech.OnInitListener() { @Override public void onInit(int status) { if (status != TextToSpeech.ERROR) { textToSpeech.setLanguage(Locale.getDefault()); textToSpeech.setPitch(1.3f); textToSpeech.setSpeechRate(1f); textToSpeech.setLanguage(Locale.UK); textToSpeech.speak(message, TextToSpeech.QUEUE_FLUSH, null, utteranceId); textToSpeech.setOnUtteranceProgressListener(new DeliveryProgressHandler(speechDialogType)); } else {

conversionDelegate.onErrorOccurred(SpeechUtil.FAILED_TO_INITILIZE_TTS_ENGI NE); } } });

return this; }

/** * Finish Speeching the Message and Shutdown */ public void finish() { if (textToSpeech != null) { textToSpeech.stop(); textToSpeech.shutdown(); }

22 Note: Full Code (Appendix G)

Usage

/* .... */ textToSpeechConverter.speakMessage("Speak Desired Building Number", SpeechDialogType.DIALOG);

/* .... */ public void cipherSpeechInput(String speechResult) { for (Building b : buildings) { if (b.getName().toLowerCase().contains(speechResult) || b.getFullName().toLowerCase().contains(speechResult)) { selectedBuilding = b; makeConfirmAlertDialog(this, "Did you say " + b.getName() + "?"); return; } else { showProgressBar(); textToSpeechConverter.speakMessage("I am sorry I didn't get that! Try again!", SpeechDialogType.DIALOG); } } }

Note: Full Code (Appendix K)

Callback

/** * Provide Results Callback from Speech To Text or Text to Speech * * @author Hardeep Singh ([email protected]) * December 28,2018 */ public interface ConversionDelegate {

void onSuccess(ArrayList result);

void onCompletion(SpeechDialogType speechDialogType);

void onErrorOccurred(String errorMessage); }

Note: Full Code (Appendix H)

23 /** * Handle Result for Custom Speech Recognition * Text To Speech Return Result on onCompletion() * Speech To Text Return Result on onSuccess() * * @param results */ @Override public void onSuccess(ArrayList results) { handleAlertDialogSpeechResults(results); }

@Override public void onCompletion(SpeechDialogType speechDialogType) { hideProgressBar(); switch (speechDialogType) { case DIALOG:

speechToTextConverter.speechWithDialog(REQ_CODE_SPEECH_INPUT_W_DIALOG); break; case WITHOUT_DIALOG: speechToTextConverter.startListening(speechRecognizer); break; } }

@Override public void onErrorOccurred(String errorMessage) { Log.e("SpeechRecognition: ", errorMessage); }

Note: Full Code (Appendix K)

4.2.5 GOOGLE LOCATION UPDATES

User location updates are required to update user location as user moves on path from origin to destination. Android devices always notifies Google with their current location.

However, these location updates cannot be accessed from the device without Google API.

Google play services integrated with application provide access to all the location-based

API’s from google. One of these API contains a generic class,

FusedLocationProviderClient. (Implementation) This client observes user device movement and notifies the application using callbacks. To handle these location update task, GoogleLocationEngine class was designed with appropriate callbacks to send data back to requesting screen. (Callback) Once user location is sent to callbacks, user location is updated on map and appropriate marking are drawn accordingly. (Usage)

24 Implementation

/** * Google Location Provider Engine * * @author Hardeep Singh ([email protected]) * March 28,2018 */ public class GoogleLocationEngine {

private FusedLocationProviderClient fusedLocationProviderClient; private Context context;

public GoogleLocationEngine(Context context) { this.context = context; fusedLocationProviderClient = LocationServices.getFusedLocationProviderClient(context); }

@SuppressLint("MissingPermission") public void startLocationUpdate(final LocationUpdateListener locationUpdateListener) { LocationRequest request = LocationRequest.create() .setPriority(LocationRequest.PRIORITY_HIGH_ACCURACY) .setInterval(300) .setFastestInterval(300);

fusedLocationProviderClient.requestLocationUpdates(request, new LocationDelegate(locationUpdateListener), null); }

public void stopLocationUpdates(LocationUpdateListener locationUpdateListener) { fusedLocationProviderClient.removeLocationUpdates(new LocationDelegate(locationUpdateListener)); }

public interface LocationUpdateListener { void onLocationUpdate(Location location); }

public class LocationDelegate extends LocationCallback {

private LocationUpdateListener locationUpdateListener;

public LocationDelegate(LocationUpdateListener locationUpdateListener) { this.locationUpdateListener = locationUpdateListener; }

@Override public void onLocationResult(LocationResult locationResult) { super.onLocationResult(locationResult);

locationUpdateListener.onLocationUpdate(locationResult.getLastLocation()); } } }

25 Note: Full Code (Appendix E)

Usage

/** * Location Updates from FusedLocationProvider * Initially only one location update is captured till user start the transit * * @param location */ @Override public void onLocationUpdate(Location location) { //Store user last location for Marker lastLocation = location;

//Setup Map Extracting User Location (Origin) and Building Location (Destination) if (startLocation == null) { setUpWithInitialLocation(location); }

//If InProgress and not simulating, Track user location and update marker if (inProgress && !simulate) { LatLng latLng = new LatLng(location.getLatitude(), location.getLongitude()); updateMarker(latLng); destinationReached(latLng); createUpdateTransitPolyLine(latLng);

//Upda D io d Di Note: Full Code (Appendix L)

Callback

/** * Callback to update User Location */ public interface LocationUpdateListener { void onLocationUpdate(Location location); }

Note: Full Code (Appendix E)

4.2.6 GOOGLE MAPS PATH PLANNING

Once user make a building selection, a path needed to be generated from user location to destination. Currently, this path is generated by Google Maps Location API. API sends back a complete set of waypoints from origin to destination in form of Polyline. Polyline

26 is line from origin to destination with all the turns and waypoints embedded into it. To extract waypoints from this polyline an algorithm need to be implemented, which can be used to draw a custom line on the map. (Usage) API call and extraction of waypoints is abstract out in a custom class, GMPath. (Implementation) GMPath is responsible of making the request to API and decoding waypoints from polyline. Once finished a callback, ResponseInterface, is used to send all this data back to requesting screen/activity. (Callback)

Implementation

public void getDirectionJSON(Context context, LatLng origin, LatLng destination, final ResponseInterface callback) { String url = generateURL(origin, destination); Log.i("Maps: ", "Direction URL: " + url); final JsonObjectRequest jsObjRequest = new JsonObjectRequest(Request.Method.GET, url, null, new Response.Listener() { @Override public void onResponse(JSONObject response) { // Pass Data Back to Caller callback.onDataReceived(response); } }, new Response.ErrorListener() { @Override public void onErrorResponse(VolleyError error) { Log.e("GMPATH VolleyException:", "Error making Volley Request: " + error.getMessage()); } }); VolleySingleton.getInstance(context).addToRequestQueue(jsObjRequest); }

Note: Full Code (Appendix B)

27 public ArrayList>> parseDirectionJSON(JSONObject jsonObject) { ArrayList>> routes = new ArrayList<>(); Log.d("Hsing", jsonObject.toString()); JSONArray jRoutes, jLegs, jSteps; try { jRoutes = jsonObject.getJSONArray("routes"); /** Traversing all routes */ for (int i = 0; i < jRoutes.length(); i++) { jLegs = ((JSONObject) jRoutes.get(i)).getJSONArray("legs"); ArrayList path = new ArrayList<>();

/** Traversing all legs */ for (int j = 0; j < jLegs.length(); j++) { jSteps = ((JSONObject) jLegs.get(j)).getJSONArray("steps");

/** Traversing all steps */ for (int k = 0; k < jSteps.length(); k++) { String polyline; polyline = (String) ((JSONObject) ((JSONObject) jSteps.get(k)).get("polyline")).get("points"); ArrayList list = decodePoly(polyline);

/** Traversing all points */ for (int l = 0; l < list.size(); l++) { HashMap hm = new HashMap<>(); hm.put("lat", Double.toString((list.get(l)).latitude)); hm.put("lng", Double.toString((list.get(l)).longitude)); path.add(hm); } } routes.add(path); } } } catch (JSONException e) { Log.e("GMPATH JSONException:", "Failed parse JSON, JSONExection " + e.getMessage()); } catch (Exception e) { Log.e("GMPATH ParseException:", "Failed parse JSON, ParseException " + e.getMessage()); }

Note: Full Code (Appendix B)

28 /** * Create PolyLine Options from JsonObject of ArrayList * * @param result * @return */ public PolylineOptions generatePathPolyLine(ArrayList>> result) { ArrayList points; PolylineOptions lineOptions = null;

// Traversing through all the routes for (int i = 0; i < result.size(); i++) { points = new ArrayList<>(); lineOptions = new PolylineOptions();

// Fetching i-th route List> path = result.get(i);

// Fetching all the points in i-th route and add to polyline for (int j = 0; j < path.size(); j++) { HashMap point = path.get(j); double lat = Double.parseDouble(point.get("lat")); double lng = Double.parseDouble(point.get("lng")); LatLng position = new LatLng(lat, lng); points.add(position); } lineOptions.addAll(points); lineOptions.width(20); lineOptions.color(Color.BLUE);

//Store way points wayPoints.addAll(points); }

// Drawing polyline if (lineOptions != null) { return lineOptions; } else { Log.d("Maps:", "PolyLine MarkerOptions are empty!"); } return null; }

Note: Full Code (Appendix B)

29 Usage

/** * Get Path From Google API * @param origin * @param destination */ public void generateDirectionsGM(LatLng origin, LatLng destination) { final GMPath gmPath = new GMPath(); gmPath.getDirectionJSON(this, origin, destination, new ResponseInterface() { @Override public void onDataReceived(JSONObject jsonObject) { PolylineOptions polylineOptions = gmPath.generatePathPolyLine(gmPath.parseDirectionJSON(jsonObject)); createPathPolyOptions(gmPath.getWayPoints(), polylineOptions); } }); }

Note: Full Code (Appendix L)

Callback

/** * Response Callbacks for GM Path Data and Serve Data * * @author hardeepsingh on March 30,2018 */ public interface ResponseInterface { void onDataReceived(JSONObject jsonObject); }

Note: Full Code (Appendix D)

4.2.7 SERVER PATH PLANNING

Once user make a building selection, a path needed to be generated from user location to destination. Currently, this path is generated by Google Maps Location API. However, as future feature this path will be generated by hosted server and deliver to application in a text file. Server will send back a complete set of waypoints from origin to destination in form of file. File is line from origin to destination with all the turns and waypoints embedded into it. To extract waypoints from this file an algorithm need to be implemented, which can be used to draw a custom line on the map. (Usage) Server call

30 and extraction of waypoints is abstract out in a custom class, SRPath. (Implementation)

SRPath is responsible of making the request to server and extracting waypoints from file.

Once finished a callback, ResponseInterface, is used to send all this data back to requesting screen/activity. (Callback)

Implementation

public void getDirectionJSON(Context context, LatLng origin, LatLng destination, final ResponseInterface callback) { //TODO: Add Params as required String url = BASE_URL; final JsonObjectRequest jsonObjectRequest = new JsonObjectRequest(Request.Method.GET, url, null, new Response.Listener() { @Override public void onResponse(JSONObject response) { callback.onDataReceived(response); } }, new Response.ErrorListener() { @Override public void onErrorResponse(VolleyError error) { Log.e("SRPATH VolleyException:", "Error making Volley Request: " + error.getMessage()); } });

VolleySingleton.getInstance(context).addToRequestQueue(jsonObjectRequest); }

Note: Full Code (Appendix C)

/** * File name and Context is passed into this method because there is no server yet * TODO: Once server is up and running, only passs jsonObject and parse is accordingly for way-points and polyline * @param jsonObject @param filename @param context */ public PolylineOptions generatePathPolyLine(Context context, JSONObject jsonObject, String fileName) { PolylineOptions polylineOptions = new PolylineOptions(); wayPoints.clear(); wayPoints.addAll(OnBoardUtil.readCordinatesFromFile(context, fileName));

polylineOptions.addAll(wayPoints); polylineOptions.width(20); polylineOptions.color(Color.BLUE); return polylineOptions; }

31 Note: Full Code (Appendix B)

Usage

/** * Get Path from Server * @param origin * @param destination */ public void generateDirectionSR(LatLng origin, LatLng destination) { final SRPath srPath = new SRPath(); srPath.getDirectionJSON(this, origin, destination, new ResponseInterface() { @Override public void onDataReceived(JSONObject jsonObject) { PolylineOptions polylineOptions = srPath.generatePathPolyLine(getBaseContext(), jsonObject, wayPointFileName); createPathPolyOptions(srPath.getWayPoints(), polylineOptions); } }); }

Note: Full Code (Appendix L)

Callback

/** * Response Callbacks for GM Path Data and Serve Data * * @author hardeepsingh on March 30,2018 */ public interface ResponseInterface { void onDataReceived(JSONObject jsonObject); }

Note: Full Code (Appendix D)

4.2.8 SIMULATOR

Simulator is used to mock the movement of user location on path for demo purposes.

Simulator use waypoints arraylist to move the user marker along the path defined.

32 /** * Simulate the marker on route for demo purposes (Requires Simulate Flag) * Animation Delay is used to cause a delay between current way-point to next way-point */ private void simulateMarker() { handler = new Handler(); final long animationDelay = 1250; runnable = new Runnable() { int i = 0;

@Override public void run() { if (i < wayPoints.size()) { LatLng current = wayPoints.get(i); updateMarker(current); createUpdateTransitPolyLine(current); lastLocation = OnBoardUtil.convertLatLngToLocation(current); updateProgressBindings(); destinationReached(current);

handler.postDelayed(this, animationDelay); } i++; } }; handler.post(runnable); }

/** * Check if destination is reached by checking if current and destination distance is less than 5 meter away * @param current */ public void destinationReached(LatLng current) { if (wayPoints.size() > 0) { LatLng destination = wayPoints.get(wayPoints.size() - 1); if (SphericalUtil.computeDistanceBetween(current, destination) < 5) { AnimationUtil.showDestinationPanel(binding); binding.bName.setText(building.getFullName()); binding.bDescription.setText(building.getSurroundings()); } } }

Note: Full Code (Appendix L)

33 CHAPTER 5

DATA FLOWS

5.1 GOOGLE MAP PATH WITH ACTIVE LOCATION UPDATES

Figure 2. Data flow with Google Maps path and active location updates

34 5.2 GOOGLE MAP PATH WITH SIMULATED LOCATION UPDATES

Figure 3. Data flow with Google Maps path and simulated location updates

35 5.3 FILE PROVIDED PATH WITH ACTIVE LOCATION UPDATES

Figure 4. Data flow with way-point file path and active location updates

36 5.4 FILE PROVIDED PATH WITH SIMULATED LOCATION UPDATES

Figure 5. Data flow with way-point file path and simulated location updates

37 CHAPTER 6

RESULTS

There will be two major evaluations of quality. First, accuracy of speech recognition model to interrupt verbal commands. Given google voice recognition is based on neural network, evaluation can be based round mean square error to find the accuracy of speech model.

However, this is only possible on server and access is permitted to server. Ultimately to evaluate google voice recognition, evaluation must be based on google yearly statics documentation. On other hand, evaluation of accurate path plotting, and path generation can be evaluated by selecting a destination and following google map poly line for accuracy. Poly line will display the full path the vehicle will have to cover to get to destination. If correct path is calculated, the polylines will follow all guidelines and all within bounds.

6.1 SELECTION/MAIN SCREEN

Application starts with selection screen that provides multiple type inputs and selection options

Figure 6. Main screen of application

6.1.1 OPTIONS

On main screen, user must select a building before a transit can start. There are multiple ways to select a building, verbally or by touch. There are multiple options like search for building, verbally pronounce the building number to make selection or scroll through the building list to select.

Search icon or magnifying glass is located on top right-hand corner of the screen. Once clicked, “Welcome to Cal Poly Pomona Transit” will be replaced with text input field and

39 keyboard will be shown. As user type the building number, building list will be updated to show only similar or exact building numbers. Once done user can tap arrow icon located on left corner of screen to go back to initial setup.

Figure 7. Search Bar usage

40 Speech

User can invoke speech listening dialog by pressing the mic on top right-hand corner.

Computer will verbally ask you to speak a building number and show a prompt waiting for your input

Figure 8. Default Google speech recognition dialog

Once user speak the building number, a voice recognition will generate a confirmation dialog with interpreted building number. If building number was not understood or does not exist in list then a vocal message will be delivered, “I am sorry, I didn’t catch that please try again!” and dialog will be shown again for next input.

Figure 9. A confirmation dialog to confirm user selection by speech or touch

Once confirmation dialog is shown a verbal message will be delivered again to confirm your entry. Voice recognition will wait for your response, to confirm the dialog with

“yes” or “no”. If input is not understood then a vocal message will be delivered, “I am sorry, I didn’t catch that please try again!” and dialog will be shown again for next input.

If yes, initial screen will be updated to portray the selection info in the bottom panel and map will mark the selected building location with marker

Figure 10. Map view when building is selected

Figure 11. Bottom panel when building is selected

If no, initial screen will not update, and original state will be shown

Figure 12. Map view when building is not selected

Figure 13. Bottom panel when building is not selected

Selection By Touch

If user choose to not use the verbal communication option. They will have the flexibility to choose from scrollable list

Figure 14. Building information list view

Upon selection, a confirmation dialog with interpreted building number with confirmation options.

Figure 15. A confirmation dialog to confirm user selection by speech or touch

Once confirmation dialog shown, user will select “yes” or “no” from confirmation dialog.

If yes, initial screen will be updated to portray the selection info in the bottom panel and map will mark the selected building location with marker

Figure 16. Map view when building is selected

Figure 17. Bottom panel when building is selected

If no, initial screen will not update, and original state will be shown

Figure 18. Map view when building is not selected

Figure 19. Bottom panel when building is not selected

Start Over

Start over button is to clear any selection and move back to original setup

Figure 20. Bottom panel building selected (before start over action)

Figure 21. Bottom panel with start over action

Route

Once building is selected, user can use the route button to choose type of transit. There are four types of transit in this application. Each transit opens a new activity with specified features. If building selection is not made and user try to select route button a message will be shown, stating “Please selection a building”. Next screen cannot be launched without a destination address (building info).

There are 4 type of transit options in this application. To allow user a choice, a dialog will be shown with all transit options. Upon selection user will be redirect to a new screen with features reflecting user selection

Figure 22. Different transit options

47 6.2 NAVIGATION SCREEN

Navigation screen handles path generation, user location updates and user transit information. As user moves on the path user location is updates to provide user a perspective of progress toward the route. User will also be provided with distance and time remaining till distance. Once distance is reached a destination panel will be launched to allow user to select new destinations.

6.2.1 TRANSIT

Launch Panel

Launch Panel shows information about user destination. It shows building number, name, description and surroundings, along with duration and time to destination. Launch panel also places two buttons on bottom of right of the screen. One button starts the transit

“Lets go”, and other button take user back to selection screen for new selection “Restart”.

Start button also zoom on to user location to provide closer view of movement as user move on the path.

Figure 23. Launch panel

Figure 24. Launch screen action buttons

In-Route Panel

In-route panel shows information take away all the building related information and only shows remaining duration and distance to the destination. In route panel also replace launch panel buttons with “Stop” button. This stop the transit, bring back the launch panel and zoom out of the user location to show full path on the map.

Figure 25. In-route panel

Figure 26. In-route action button

Destination Panel

Once destination is reached a destination panel is shown replacing in-route panel.

Destination panel takes away all the side panels and show a panel on bottom screen with destination surrounding and a restart button to select a new destination.

Figure 27. Destination arrived panel with action button

50 Google map path generation with active location updates

Transit with google path generation use Google location API to generate the path from user current location to user selected destination. Once path is generated, map will automatically zoom-in on the path, highlight the path with blue line and present a launch panel to user for desired actions. In this transit, once user start the transit, location will be updated as user move, and green line will be drawn on the map to represent user movement.

Figure 28. Google map path and active location updates, launch screen

Figure 29. Google map path and active location updates, in-route screen

51 Google map path generation with simulated location updates

Transit with google path generation use Google location API to generate the path from user current location to user selected destination. Once path is generated, map will automatically zoom-in on the path, highlight the path with blue line and present a launch panel to user for desired actions. In this transit, once user start the transit, location will be updated by simulator (mock user movement), and green line will be drawn on the map to represent user movement.

Figure 30. Google map path and simulated location updates, launch screen

Figure 31. Google map path and simulated location updates, in-route screen

52 Way-point file Path Generation with active location updates

Transit with way-point file path generation will use provided waypoints file to generate a path to destination. Once path is generated, map will automatically zoom-in on the path, highlight the path with blue line and present a launch panel to user for desired actions. In this transit, once user start the transit, location will be updated as user move, and green line will be drawn on the map to represent user movement.

Figure 32. Way-point file path and active location updates, launch screen

Figure 33. Way-point file path and active location updates, in-route screen

Note: Distance and duration information might not be correct because screenshots were taken away from university. Hence estimates reflect my actual location calculations

53 Way-point file path generation with simulated location updates

Transit with way-point file path generation will use provided waypoints file to generate a path to destination. Once path is generated, map will automatically zoom-in on the path, highlight the path with blue line and present a launch panel to user for desired actions. In this transit, once user start the transit, location will be updated by simulator (mock user movement), and green line will be drawn on the map to represent user movement.

Figure 34. Way-point file path and simulated location updates, launch screen

Figure 35. Way-point file path and simulated location updates, in-route screen

Figure 36. Destination reached panel Project satisfied all project goals by integrating everything in a neat and animates user interface. Application can follow user on path and simulate the effect of user moving on a path, allowing it to be a complete product for production or future developments.

55 CHAPTER 7

EVALUATION OF RESULTS

To evaluate results, error between given path and user movement on that path, is calculated. However, path consists of less waypoints compare to user movement. User movement is logged every 700 milliseconds while moving along path. However, path waypoints density increases or decrease according to turns and straight away. This raise an issue for RMSE (Root Mean Square Error) calculation because there is no 1 to 1 ratio between two datasets.

Equation 2. Root Mean Square Error equation

7.1 APPROACH

To create 1:1 ratio between two datasets, an algorithm was developed which maps closest path point to user location.

//Find closest Location GEO points from Path file and store it val geoPointInfo = mutableListOf() locationList.forEach({ val location = it var shortestDistance = Double.MAX_VALUE var shortDistancePoint: GeoPoint? = null pathList.forEach({ val distance = getDistanceInMeters(location, it) if (shortestDistance > distance) { shortestDistance = distance shortDistancePoint = it } }) geoPointInfo.add(GeoPointInfo(shortDistancePoint!!, location, "%.5f".format(shortestDistance).toDouble(), null)) })

Note: Full Code (Appendix M)

56 This create a new set of data with same path waypoint pointing to multiple user location

(GeoPointInfo). To overcome any unnecessary data, an average of all the location waypoints were calculated associated with single path waypoint (DistinctGeoPointInfo).

var distinctGeoPointInfo = mutableListOf() pathList.forEach({ val pathPoint = it var averageLat = 0.0 var averageLng = 0.0 var dataPoint: GeoPointInfo? = null var counter = 0 geoPointInfo.forEach({ dataPoint = it if (pathPoint == it.pathGeoPoint) { averageLat += it.locationGeoPoint.latitude averageLng += it.locationGeoPoint.longitude counter++ } }) if (averageLat != 0.0 && averageLng != 0.0) { val averageLocationGeoPoint = GeoPoint(averageLat / counter, averageLng / counter) val newGeoInfoPointInfo = GeoPointInfo(pathPoint, averageLocationGeoPoint, dataPoint!!.distance, GeoPoint(it.latitude - averageLocationGeoPoint.latitude, it.longitude - averageLocationGeoPoint.longitude)) distinctGeoPointInfo.add(newGeoInfoPointInfo) } })

Note: Full Code (Appendix M)

As a result, 1:1 data set was created for path and user location waypoints. An equation of

RMSE is applied to latitude and longitude to get an error for each property.

fun rootMeanSquareError(data: List): List { var totalLat = 0.0 var totalLng = 0.0 var totalDistance = 0.0 var n = data.size data.forEach({ totalLat += Math.pow(it.residual!!.latitude, 2.0) totalLng += Math.pow(it.residual!!.latitude, 2.0) totalDistance += Math.pow(it.distance, 2.0) })

return listOf(Math.sqrt(totalLat / n), Math.sqrt(totalLng / n), Math.sqrt(totalDistance / n)) }

Note: Full Code (Appendix M)

57 7.2 RESULTS

Compute o Building 17:

Root Mean Square Error Latitude: 4.575669127001064E-5 Root Mean Square Error Longitude: 4.575669127001064E-5 Root Mean Square Error Distance: 5.357289999999999

Note: Detailed Data Set (Appendix N)

Compute to Building 66:

Root Mean Square Error Latitude: 6.720915119232342E-5 Root Mean Square Error Longitude: 6.720915119232342E-5 Root Mean Square Error Distance: 7.113816000000002

Note: Detailed Data Set (Appendix O)

Compute to Building 77:

Root Mean Square Error Latitude: 5.979131832767509E-5 Root Mean Square Error Longitude: 5.979131832767509E-5 Root Mean Square Error Distance: 6.1056239999999974

Note: Detailed Data (Appendix P)

Compute to Building 112:

Root Mean Square Error Latitude: 4.280093507626858E-5 Root Mean Square Error Longitude: 4.280093507626858E-5 Root Mean Square Error Distance: 4.1991999999999985

Note: Detailed Data Set (Appendix Q)

58 CHAPTER 8

FUTURE WORKS

8.1 ENHANCED SPEECH RECOGNITION

Given android application is speech driven, speech recognition model has to be trained for specific building number. For example, “Building 1” get deciphered as “Building 1” but “Building 12b” get deciphered as “Building Twelve B”. This make it very difficult to understand user command against the actual building data source. To improve on this functionality, a custom speech recognition engine need to be implemented with custom speech model to decipher user input correct

59 REFERENCES

Adams, Chris. "Explaining the Different Types of Voice Recognition." ThoughtCo.

N.p., 30 June 2013. Web. 06 July 2017.

Churchill, Elizabeth, Anne Bowser, and Jennifer Preece. "Interactions." Teaching and

Learning Human-computer Interaction. IX Interactions, Mar.-Apr. 2013. Web.

06 July 2017.

Google "Google Maps Type and Usages." Google. Google, 7 June 2017. Web. 07 July

2017.

Kalyanpur, Aditya, and J. William Murdock. "Unsupervised Entity-Relation Analysis in

IBM Watson." Proceedings of the Third Annual Conference on Advances in

Cognitive Systems. Cognitive System, 2015. Web. 23 Feb. 2017.

Kunal Shrivastava, Nishant Singhal, Pradip K. Das, and Shivashankar B. Nair. 2013. A

Speech Recognition Client-Server Model for Control of Multiple Robots. In

Proceedings of Conference on Advances In Robotics (AIR '13). ACM, New York,

NY, USA, , Article 69 , 6 pages

Madrigal, Alexis C. "How Google Builds Its Maps—and What It Means for the Future of

Everything." The Atlantic. Atlantic Media Company, 06 Sept. 2012. Web. 07 July

2017.

Stanley, Morgan. "Autonomous Cars Self-Driving the New Auto Industry Paradigm."

(n.d.): n. pag. Princeton. Princeton Education, 6 Nov. 2013. Web. 23 Feb. 2017.

Stephanidis, Constantine. "1." Universal Access in Human-Computer Interaction.

Mahwah, NJ: Lawrence Erlbaum Associates, 2001. 3+. Print.

Young, Richard M., and Phil Barnard. "The Use of Scenarios in Human-Computer

60 Interaction Research: Turbocharging the Tortoise of Cumulative Science." MRC

Applied Psychology Unit, Mar.-Apr. 1987. Web. 14 Mar. 2017.

61 APPENDIX

A. BuildingInfo