Wikitude { Usability testing and heuristic evaluation O. Perutkaa aFaculty of Information Technology, CTU, Prague, Czech Republic Abstract Since augmented reality mobile applications depend on surrounding environment, usability testing in a laboratory environment is not always applicable for them. In addition, expert usability evaluation can be used to discover general usability problems which may be ignored by test subjects. This paper describes modified heuristic evaluation and a real environment usability testing method applicable for augmented reality navigation applications. Both methods are applied on application called Wikitude. Results of these evaluation methods are presented. Keywords: Wikitude, navigation, augmented reality, usability testing, heuristic evaluation. 1. Introduction and testing in real environment by observing. The heuristic evaluation is targeted to discover general Augmented reality (AR) applications for mobile usability problems that might be ignored by real devices became popular in a last few years. Mostly users but are important for their overall satisfac- thanks to the growing market with highly available tion. The real environment testing should discover smart devices. Standard usability evaluation tech- common user habits and patterns of using the ap- niques are not always suitable for these applications plication. It should verify the efficiency of using because of their often context dependency. This pa- this application. per starts with a brief description of a representa- Tables A.3 and A.4 contain an adaptation of the tive of these applications { Wikitude. In the follow- ing sections there is a description of the used evalua- original Jakob Nielsen's heuristic evaluation [3]. It tion methods, their modifications, testing scenarios has been modified for purpose of evaluation of the and test results. The usability testing was inspired Wikitude user interface. by [1] and [2]. Two testing scenarios for testing in a real envi- Wikitude is an AR navigation for Android plat- ronment were prepared for a representative group of form. It has many categories of points of interest databases and/or points of interest categories (see (e.g. sights, hotels, restaurants, shops etc.). The ap- below). They cover neighborhood discovery (e.g. plication allows users to list a set of nearby points sightseeing, looking for a restaurant etc.) as well as of interest (list-view), show them on the top of a searching for concrete points of interest (e.g. shops, local map (map-view) and show them as labels on bus stops etc.). The goal of this testing was to an- the top of a live video feed (live-view). There is also alyze overall efficiency of use and usage patterns. a search functionality which allows to search for a Data from testing were recorded in form of written particular point of interest. The application de- notes. pends on external web services such as Wikipedia, Objective of the first scenario was to compare ef- Yelp.com and Hotels.com. It uses GPS and mobile ficiency of Wikitude and printed tourist guides and networks to determine user's location and electronic maps (publicly available at airports, railway sta- compass to get user's orientation. tions etc.). The task was divided into two parts. In the first part the test subject had five minutes to go through a given set of tourist information in 2. Evaluation paper form (see below) and another five minutes Two methods have been chosen for purpose of us- for using Wikitude. The order of these two ses- ability evaluation of Wikitude { heuristic evaluation sions was randomized by flipping a coin. Each time Preprint submitted to Elsevier April 24, 2014 the test subject had to find as many sights in a 3. Number of mistakes done while navigating us- reasonable distance (approximately 2 km) as pos- ing the printed map and using Wikitude sible. The initial position of the test subject was 4. Number of times the test subject consulted the at Central railway station in Helsinki. In the next map or Wikitude while navigating from one part the test subject went from the Central rail- point to another way station to the Helsinki Cathedral and from the Helsinki Cathedral to the Uspenski Cathedral (ap- The following resources in paper form were avail- proximately same walking distance in both cases). able (both taken from the Central railway station): In both cases he had to use both { the printed map 1. Helsinki City Tourist Map by City-Opas and Wikitude. The printed map was used for one 2. Helsinki Visitors Guide half of the path and Wikitude was used for the other half. The order was also randomized by flipping Objective of the second scenario was to deter- a coin. The order for the path from the Helsinki mine efficiency of Wikitude search. The test subject Cathedral to the Uspenski Cathedral was reversed was intended to find the nearest stationery. The (i.e. since the order for the path from the Central subject's initial location was Kumpula Campus (in railway station to the Helsinki Cathedral was: first front of the Physicum building). The evaluation use the printed map and then Wikitude, the order metric was number of search results explored be- for the second path was: first use Wikitude and then fore finding the required result. the printed map). These are the exact instructions for the test sub- These are the exact instructions for the test sub- ject (they were presented in oral form): ject (they were presented in oral form): 1. Take this cell phone and open Wikitude. Use 1. Take this map and this tourist guide and try the search bar to find the nearest stationery. to find as many sights as possible. The sights Imagine you want to buy a notebook and a should not be further than approximately 2 km pen. Tell me each time you access a search (current location was shown). Tell me each result to get a detailed info until you find a time you find something. You have five min- shop where you can buy these things. utes. 2. Take this cell phone and open Wikitude. Try to find as many sights as you can. They should 3. Results of the heuristic evaluation not be further than approximately 2 km. Tell Violations of the rule 1. The application should me each time you find something. You have always keep user informed about what is going on. five minutes. The information about the used database or a cat- 3. Take this map and go to the Helsinki Cathedral egory of points of interest should be always visible (current location and location of the cathedral as well as location tracking status and orientation. were shown). All information should be given through appropriate 4. After one half of the path { Give me the map feedback within reasonable time. and use Wikitude for the rest of the path (lo- cation of the cathedral was preset). 1. The graphic symbol used to show user's current 5. Go to the Uspenski Cathedral, use Wikitude location in the map-view can be covered by for navigation (location of the cathedral was labels related to points of interest (see Figure preset). 1). It is hard to find the user's location in such 6. After one half of the path { Give me the cell case. phone and use the map for the rest of the path 2. The application shows only camera live-view (current location and location of the cathedral after selecting some categories of points of in- were shown). terest (e.g. Hotels.com or Yelp.com). There are The evaluation metrics for this experiment were: no controls or system status visible. 3. Very slow response times in some cases (mostly 1. Number of sights discovered using the printed while switching between various categories of tourist information and using Wikitude points of interest). 2. Amount of time required for navigating from one point to another using the printed map Violations of the rule 8. Screens should not con- and using Wikitude tain information which is irrelevant or unneeded. 2 Figure 1: Location indicator covered by labels and uncovered after zooming. 1. There are buttons for following the application scenario. Surprisingly, the difference between the on Facebook and Twitter mixed into featured numbers of discovered sights is not big. There were points of interest categories. It may be confus- basically three reasons: ing. 1. Not all results shown by Wikitude could be Violations of the rule 9. All errors should be considered as sights from a tourist point of in a clear, user-readable form and they should sug- view. gest a reasonable solution. If an error happens, user should be notified. There should be no \silent" er- 2. There were some duplicities between the re- rors. sults (same sight, different name or differ- ent language mutation of a corresponding 1. The application stops responding randomly Wikipedia page). when switching between live-view, list-view 3. Wikitude has limited amount of points of in- and map-view. This happens also after return- terest it can show. The maximal amount is 50, ing from idle mode. so the application shows 50 closest sights. It Violations of the rule 10. Provide some help is hard to find more distant and possibly in- and/or documentation. Both a quick start help and teresting sights in areas with high density of a detailed documentation should be available. sights (such as Helsinki city center). 1. There is no help. The provided demo does not Another interesting observation is that the sets of work. In addition its usage requires to open a sights discovered using the printed map and sights demo web page on a screen of another device discovered using Wikitude are almost disjoint.