Validation of Navigation Guidelines for Improving Usability in the Mobile Web

Home , Web navigation

Eva Garcia-Lopez*, 1, Antonio Garcia-Cabot1, Cristina Manresa-Yee2, Luis de-Marcos1, Carmen Pages-Arevalo1

1Computer Science Department, University of Alcala, Alcalá de Henares, Spain.

2Maths and Computer Science Department, University of Balearic Islands, Palma de Mallorca, Spain.

1Dpto Ciencias Computación. Edificio Politécnico. Ctra Barcelona km 33.1. 28871. Alcalá de Henares. Madrid. Spain.

2Ed. Anselm Turmeda. Ctra Valldemossa km 7.5. 07122. Palma de Mallorca. Islas Baleares. Spain.

1Phone: +34 918856836

*Corresponding author: [email protected]

Validation of navigation guidelines for improving usability in the mobile web

Abstract: Traditional navigation guidelines for desktop computers may not be

valid for mobile devices because they have different characteristics such as

smaller screens and keyboards. This paper aims to check whether navigation

guidelines for Personal Computers are valid or not for mobile devices. To do

this, firstly an evaluation was carried out by a group of advisors with knowledge

in web design, and then empirical studies were conducted with real mobile

devices and users. Results show that most of the traditional navigation

guidelines are valid for current mobile devices, but some of them are not, which

suggests that a similar process is necessary to validate traditional guidelines for

mobile devices. The final result is a set of fifteen navigation guidelines for

mobile web sites, from which eleven are totally new because they have not been

previously suggested by any standard or recommendation for mobile web sites.

Keywords: hyperlink; navigation; mobile device; usability guideline; experimentation

1. Introduction

Hyperlinks, also known as anchors or HTML tags, are the primary mode of navigation on the web [1]. They connect web objects in a web site and allow users to navigate from the current page to another site on the Internet [2, 3], thus allowing web designers to organize the data on web pages in different ways [4], according to the main purpose of the site or the target audience.

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. Different techniques have been proposed for linking elements between web pages [5-7], but the most discussed have been access keys [8]. However, this method cannot be used on touchscreen mobile devices due to the fact that shortcuts are implemented using a

XHTML attribute (called ‘accesskey’) that only works with 0-9, * and # keys of hardware keyboards [9], so they can only be used in a few current mobile browsers.

However, when they work in a mobile browser, this alternative can be used in combination with hyperlinks because it can provide shortcuts to expert users, but in this case they have to be designed in such a way that they do not confuse non-expert users and encourage number pressing [5]. Anyway, users should always be able to navigate with hyperlinks when they are using mobile devices [5], and hyperlinks have been proved to be the easiest way to navigate in mobile web sites [6], so using them should be enough and more attention should be paid to their usability.

Hyperlinks are one of the top 10 key factors influencing webpage design [10], and represent about 50% of web interactions [11]. On the other hand, the number of web users is increasing, as well as the number of mobile users, so studying the usability of hyperlinks is crucial in order to facilitate the Internet navigation, especially through these devices. According to Nakamoto [12], hyperlinks are highly suitable for improving user interfaces in smart devices, but studies carried out for Personal

Computer (PC) interfaces about hyperlinks may not be applicable to this kind of devices because the number of hyperlinks displayed on a desktop PC within a single screen or page is higher than the number of hyperlinks that can be displayed on a mobile device

[13]. Because of this, it is necessary to study the usability of hyperlinks in mobile devices separately from the usability of hyperlinks in PCs.

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. There are two main factors to be studied about hyperlinks, which are complementary and influence the efficiency of navigation and, therefore, their usability: the design of links and the structure of links [14-16]. The former includes the appearance of links, such as their style or location on the web pages, as well as their content [17]; and the latter refers to how hyperlinks connect the documents on the web [18]. In this paper we focus on the visual aspects of the hyperlinks, so the literature review will address this topic.

This paper shows a strict process of analysis and evaluation to contribute with guidelines for designing hyperlinks in mobile devices in order to improve their usability. This process consists of (1) an evaluation of usability guidelines for detecting which of the traditional navigation guidelines could be applied to mobile devices; and

(2) empirical experimentation for validating these guidelines and proposing new or adapted navigation guidelines for mobile browsers.

Section 2 presents the related work to this paper, Section 3 describes the first stage of the method carried out (i.e., selection of guidelines), and Section 4 explains the experimentation and results. Section 5 proposes the new guidelines about hyperlinks for mobile devices, Section 6 shows a discussion about our results compared to prior research and finally Section 7 exhibits the conclusions.

2. Background

Usability of PC interfaces has been widely studied. There are published studies about different aspects such as formatting, contrast and color [19], line length [20], pictograms and icons [21] or scrolling [22]. However, the differences between PCs and mobile devices makes the interaction of both devices different, so usability may be influenced

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. by the type of device used. This was suggested by several studies, which indicate that the small screen size of mobile devices may negatively influence effectiveness [23], efficiency [23-27] and satisfaction [28], which are in turn the metrics included in the definition of the term “usability” by ISO [29].

This section aims at compiling the guidelines and previous work concerning the visual aspects of the links, and more specifically hyperlinks for mobile devices. It is important to note that the term “guideline” has been considered here as a rule about what to do or not to do, given by an official organization or by a study that has validated it through experimentation. Most of the existing usability guidelines for hyperlinks have been designed for PC interfaces, but usability of hyperlinks in mobile devices should be specifically studied, as some studies demonstrated in other areas [30, 31]. In the following paragraphs, there is a compilation of guidelines for designing hyperlinks, retrieved from different studies, standards and recommendations. These guidelines were retrieved from different standards and documentation from official organizations such as the U.S. Department of Health and Human Services, the World Wide Web

Consortium (W3C) and the International Organization for Standardization (ISO). Four documents of this type were found: the HHS Guidelines [32], ISO 9251-151 [33], W3C

Mobile Web Best Practices [34] and WCAG 2.0 [35], which was accepted as ISO

40500. Furthermore, different search engines (Google Scholar, ScienceDirect, ACM digital library and IEEE Xplore Digital Library) were used for finding research reports containing guidelines obtained through experimentation. Only one study was found after this process: the Usability of Mobile Websites and Applications by Budiu and

Nielsen [36]. The guidelines about hyperlinks found in those standards and recommendations are summarized in Table 1.

The main problem with hyperlinks is that their names (or labels, i.e. the text) do not always clearly inform about the content of the target web page [37]. When link descriptions are ambiguous or unclear, users try various links before they can find what they need, thus users do not get the information efficiently [38]. Therefore, the hyperlink names should be meaningful, understandable, unambiguous, descriptive and they should clearly tell what is the content that will be reached [2, 32, 33, 39], and more specifically the text of the link should be the same as the title of the target web page [5,

32]. In general, generic instructions such as “Go”, “Click here”, “More…” or “Next article” should be avoided [33, 36, 40]. In addition, according to Sprengers [39], not only the text of the hyperlink but also the surrounding text must relate to the topic of the hyperlink so that users are given a good sense of what they should and should not expect from clicking on the hyperlink. The guideline about clearly identifying the target of each hyperlink is even more important for mobile devices [36], because it may avoid or reduce delays resulting from following incorrect links [34].

Hyperlinks can link objects (called “special targets”) other than web pages, i.e. they can open documents [2], such as PDF or word processor documents, e-mail messages or other type of files that are opened with different applications. In this case, the hyperlink should explicitly indicate it [33, 40]. When a file is exceptionally large or the target of the link is in a different language, an appropriate indication should be included to inform about the special target type [33]. This is especially important for mobile devices

[34], since downloading a large file could increase the cost and some types of file formats may not be opened when using this kind of devices.

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. Another relevant topic about hyperlinks is consistency. Important hyperlinks can be repeated (as needed) elsewhere on the web page or site to ensure that important content can be accessed from different paths [32], but their format should be consistent throughout all pages [5, 33], as well as the text of the redundant hyperlinks [33].

Currently the authors have not found usability guidelines about this topic for mobile devices, although this aspect does not seem to be related to the screen size or the interaction method.

When a hyperlink is next to another, they should be visually separated, e.g. by non-link printable characters [33]. This is one of the methods to facilitate the identification of links, which is a very common guideline, but not the only one. Hyperlinks should be clearly identified, i.e., users should also be able to easily differentiate hyperlinks from regular text [2, 33, 40, 41], avoiding the design of visual elements that look like links but which are actually not links [33]. Therefore, it is necessary to highlight the hyperlinks, which can be done by changing their format. There are several techniques to facilitate the identification of links using formatting, such as using underlined text for hyperlinks and showing them in a different color [42]; using bold and bold-underlined hyperlinks [43]; or just underlining them and not underlining any text that is not a link

[39, 41, 42, 44]. Regarding the color of hyperlinks, it has been proved that blue color on a white background improves accuracy and display quality when compared to black links on a white background [45], probably because users are accustomed to this format by default in most current web pages. Visited and unvisited links should also be marked with different colors so that users are able to distinguish when they have already clicked on a link [32, 33, 40, 42]. As in the case of consistency, the authors have not found

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. usability guidelines about this topic specifically for mobile devices, but it does not seem to be related to the screen size or the interaction method.

Although a study affirmed that highlighting or not the hyperlinks does not have an impact on text comprehension [46], the readability of texts containing hyperlinks can be compromised when there are too many hyperlinks on a web site, since they may distract or impede the readability of the text [2, 33], especially if users are older adults [47, 48] or children [49]. Furthermore, when hyperlinks are too long, they may take up more than one line, but if they are too short they may not be understood [32, 33]. Therefore, avoiding hyperlink overloading is recommended whenever possible, as well as using an appropriate length, short enough to minimize wrapping and long enough to be understood. Although the last guideline might vary in mobile devices because of the small screen size (there are more line breaks), no usability guidelines have been found during the literature review about this for mobile devices.

Sometimes hyperlinks can be showed as an icon rather than text, but this is not recommended by most researchers [2, 32, 42] because textual hyperlinks are easier to use and clearer to follow for users [50]. In addition, ISO [33] says that ‘graphical symbols are only useful for common and frequently used link types and if they represent a well-known metaphor’. In the case of mobile devices, according to Budiu and Nielsen [36], using text rather than icons for hyperlinks is also preferable because images can be loaded late and cause problems on slow connections, since users may want to take an action and click on a hyperlink before the whole web page is loaded.

Finally, sometimes users could be confused when they click on a hyperlink and are forwarded to a different web site or navigation context [33, 51]. Therefore, opening new

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. windows or using the existing ones should be carried out according to users’ expectations [52]. So the last guideline found in literature is that users should be informed when a hyperlink opens its target on a different web site, a new browser window, a pop-up window or in a different location on the same page [32, 33]. This can be achieved, for example, by indicating it in the text of the hyperlink or by means of graphical representation. No guidelines about this topic for mobile devices have been found in current standards or literature.

At this point, no studies about checking the validity of traditional (PC) guidelines in mobile devices have been found. As it can be seen in Table 1, ISO 9241-151 [33] is the most complete standard because it is the one that contains more guidelines about hyperlinks. Therefore, this paper focuses on these guidelines and tries to check whether or not they would be valid for current mobile devices. This study is necessary because standards and recommendations providing guidance on increasing usability for designing web user interfaces may not be directly applicable to web mobile interfaces because of the different characteristics of mobile devices [53, 54].

Since no processes have been found to obtain mobile usability guidelines from PC guidelines, the authors have designed a new approach that consists of three stages: (1)

Selection of guidelines, (2) experimentation and (3) proposal of guidelines. The methods carried out in each stage are explained below, as well as the results found in each stage.

3. Selection of guidelines

The first stage consisted of an evaluation by a group of advisors. We defined advisors as informed individuals, specialists in a field, or those with knowledge or abilities about

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. the specific subject under study, in this case web design. The advisors had to judge whether or not each traditional guideline about hyperlinks could be valid (i.e., if it could make sense) for mobile devices. In this context, we considered a mobile device as a portable computing device such as a smartphone, which is in turn a mobile phone that performs many of the functions of a computer, typically having a touchscreen interface,

Internet access, and an operating system capable of running downloaded apps [55]. To do this, they classified the traditional guidelines (from 9.4.2 to 9.4.15 inclusive of ISO

9241-151, which refer to hyperlinks) into one of the following categories:

a) Not applicable. These guidelines are not transferable in any way to mobile

devices, i.e., they make no sense in web pages for mobile devices. This usually

happens because the guideline may require specific hardware that is not

available in mobile devices, such as a PC keyboard or a mouse.

b) Directly applicable. These guidelines would not suffer any modification when

applied to mobile devices, i.e., they would still make sense in web pages for

mobile devices.

c) Applicable with modifications. These guidelines could suffer a modification

before being applied to mobile devices, i.e., they could make sense in web pages

for mobile devices, but being previously modified.

The group of advisors was composed of 16 experts with a computer science background working in academic and/or professional fields and focused on web design, web and mobile development or usability evaluation to ensure a heterogeneous range of opinions. Table 2 summarizes sociodemographic information.

[Table 2 near here]

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. During the evaluation, experts assessed which of the traditional guidelines could be valid for mobile devices, by filling a template where they selected one of the above options (a, b or c) for each original guideline, justifying their answers. After that, a Chi- square analysis was performed for each guideline in order to determine whether or not the responses followed a random distribution. A random distribution here means that there was no a clear trend toward one of the options from the advisors. Those guidelines with a random distribution in responses as well as those whose general answer was “c)” had to be subjected to experimentation. The rest could be initially accepted by considering the most selected answer. Therefore, guidelines with majority of “b)” answers would be directly adopted for mobile devices, whereas guidelines with majority of “a)” answers would not exist for mobile devices.

Table 3 shows the results of the Chi-square analysis performed for each guideline.

[Table 3 near here]

When p-value is greater than 0.05, the answers of advisors did not follow a clear trend and therefore experimentation was needed. As seen in Table 3, there are 2 guidelines with p-value > 0.05 (cells’ background in grey).

Regarding guideline 9.4.10, 37.50% of advisors answered that it could be modified when applied to mobile devices because no new windows should be opened in this kind of devices, since it is uncomfortable; or otherwise animation effects could be used (such as the iOS’s) to indicate that the link is being opened in a new window. On the other hand, 43.75% of advisors answered that guideline 9.4.13 could not be directly applied to mobile devices because the screen of mobile devices is small and probably the number of links that can be shown in these devices is smaller than in PCs. In both cases, the

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. guideline would change for mobile devices. Therefore, experimentation was performed for those guidelines (9.4.10 and 9.4.13), as explained in Section 4. The rest of guidelines had most answers “b”, so that they would remain as they are enunciated for mobile devices.

4. Experimentation

An experiment was designed and performed for guidelines whose answers followed a random distribution (i.e. 9.4.10 and 9.4.13), in order to check whether or not the guideline would be directly applicable to a mobile device, or if it would be applicable but after some modifications. Experimentation could also be needed for guidelines where the most selected option was “c”, but this was not the case (the rest of the guidelines received a majority of “b” answers). Experimentation could be performed in different ways, depending on the guideline to be checked: they could be carried out with users and/or with real mobile devices.

Experiments with users were performed by designing at least two alternatives to be compared (meeting the guideline and not meeting it), in order to know if usability improved. For measuring usability, the metrics effectiveness, efficiency and satisfaction were used, from the “usability” definition by ISO 9241-11 [29]. In experiments involving users, a webcam was used to record the smartphone’s screen. The webcam was joined to a metal structure, and the structure was joined to a soft housing specific for this kind of device (Figure 1). Then, the webcam was connected to a laptop via the

USB cable, so videos were recorded directly into the laptop for later analysis. Videos were recorded using the software provided with the webcam. Camtasia Studio was used to review the videos and mark times when the user started and finished the experiment,

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. as well as the errors made. Marks were then used for counting errors and time taken to perform the tasks.

[Figure 1 near here]

Experiments with mobile devices should usually check if they support some functionality necessary to validate the guideline in this kind of devices, but other research questions could also be posed.

All the experiments were performed in a research lab, and participants were firstly instructed by the researcher about the tasks they had to carry out. They had to work as usual, on their own way, trying to carry out interactions as realistically as possible.

After the experiment, participants had to answer a questionnaire about the interface.

Two experiments were designed to check whether or not traditional guidelines 9.4.10

(Marking links opening new windows) and 9.4.13 (Link length) are valid for mobile devices. The design and results of the experiments are explained below.

4.1. Experiment 1: Marking links opening new windows

Guideline 9.4.10 of ISO 9241-151 states that hyperlinks opening new browser windows should be clearly marked. To check whether it is valid or not for mobile web sites, the hypothesis was “if a link opens a new browser window or a pop-up window, it should be clearly marked”. In order to accept or reject the hypothesis the experiment consisted of two parts. The first one was a preliminary study with mobile devices to check if mobile devices support multiple windows. This is necessary because, if mobile devices could not support multiple windows, the guideline would not make sense in mobile web sites because new windows would not be opened, so marking links opening new

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. windows would never be relevant for mobile devices. The second part consisted of an experiment performed with users, to check if the guideline would be valid for mobile devices as it is currently stated for PC web sites, or if it would have to be modified before applying it to mobile devices.

4.1.1. Part 1: preliminary study with mobile devices

A simple web page containing only two links was designed: one hyperlink that opened the target in the same window and another one that opened the target in a new window.

The behavior of both links was tested on twenty different mobile devices, using their default browser. Results showed that 19 out of 20 mobiles (95%) used in the test supported multiple browser windows (Table 4), thus contradicting the W3C indications

[34], probably because mobile devices have been evolving in recent years, by adding new functionalities.

[Table 4 near here]

4.1.2. Part 2: experiment with users

Since most of current mobile devices support multiple windows, an experiment was necessary to determine if the guideline would be valid for mobile web sites as it is for

PC web sites. The experiment then compared the performance of users when web pages included or not notifications about opening links in new windows while using different mobile devices.

4.1.2.1. Test setting

Two web pages were designed, each of which contained only links (there was no any other text on the web page) that opened their target by default in the same window and

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. some others in a new window. The targets of the links in the first web page were not indicated. The hyperlinks in the second web page had two different types of notification

(with text or with icon), to examine also if there is a significant difference depending on the method used [56]. The type of icon may also influence results [57], so the icon used in the official Transport for London web page1 for indicating links opening new windows was chosen because a previous study demonstrated this is a usable web page

[58].

Nineteen subjects participated in the experiment (13 male, 6 female). None of them participated as advisors in stage 1. Regarding the age distribution, nine of them were between 18 and 24 years old, seven were between 25 and 34 years, and three were older than 35 years. As to the subjects’ self-rating on their experience in using mobile devices, eight were considered experts, nine were intermediate, and two were novice.

The subjects used six different mobile devices to perform the experiment, which consisted of opening (using the default web browser) the hyperlinks in the same window or in a new window, according to what the experimenter requested. The experiment had a within-subjects design, and the order of use of the different devices as well as the link to be opened were randomized for each subject, with the aim to mitigate the order effect. Usability of each interface (notifying about the target of links or not notifying about it) was measured using effectiveness, efficiency and satisfaction metrics. Effectiveness was measured through the number of errors made and whether the user was able to correctly complete the task or not. The number of errors was the number of times the hyperlink was opened in a destination other than the requested. To

1 Visit https://tfl.gov.uk/fares-and-payments/oyster to view an example of the icon.

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. measure efficiency, the time taken by users to successfully open each link was recorded, and if a user failed, the time was measured until he/she gave up. There is a discussion in the HCI and usability literature about whether to include failed tasks or not in task completion time [59, 60], but the authors decided to include them because the number of participants was very low and otherwise it would have been very few data to analyze.

Finally, after performing the experiment, users filled out a satisfaction survey asking them about their preferences for opening links in new windows on each device.

The treatments of the experiment were randomly assigned to each user, thus mitigating possible biases introduced by other factors. All users performed the experiment once with each configuration (i.e., links without notifying about their target and links notifying about it) and on each mobile device (six different devices). Since browsers of different operating systems have different ways of interaction, the results could be biased if we had used only one device. Therefore, five mobile devices were selected to include the most used mobile operating systems: iOS, Windows Phone, Android,

BlackBerry and Symbian. Furthermore, since hardware may also influence the interaction, one more mobile device was selected, which had hard keyboard. All possible combinations were tested: on the one hand, a link that opens by default in the same window had to be opened (1) in the same window and (2) in a new window; and on the other hand, a link that by default opens in a new window also had to be opened

(3) in the same window and (4) in a new window. These combinations were chosen because web pages have links that by default open both in the same and in a new window, but users may want to open them in the same or in a new window according to their preferences at any time. In total, 48 videos (2 configurations x 6 devices x 4

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. combinations = 48) were recorded for each user, and later analyzed. After the experiment, each user answered the satisfaction survey.

4.1.2.2. Methodology

Effectiveness and efficiency were measured considering: (1) task completion, (2) number of errors and (3) the time taken to perform the task in the following cases:

 Open in the same window a link that by default opens in the same window (to

simplify, from now on this operation will be called “Same-same”).

 Open in a new window a link that by default opens in the same window (“New-

same”).

 Open in the same window a link that by default opens in a new window (“Same-

new”).

 Open in a new window a link that opens by default in a new window (“New-

new”).

In addition, for the cases in which a hyperlink opened by default in a new window (i.e.,

“Same-new” and “New-new”), the possible effect of informing with a text and informing with an icon was also studied. Figure 2 shows all cases for each user.

[Figure 2 near here]

As aforementioned, the subjects filled out a satisfaction survey after the experiment.

Standard questionnaires such as SUS [61] or SUMI [62] were not used because they are very generic, and authors included another specific questions. According to Hornbæk

[63], satisfaction can be measured as attitudes towards the interface, and that is what we

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. intended to study. The survey contained the following questions with predefined answers to be selected:

Q1. Knowing that the target of a link will open in a new window, when would you

click on it? [“If I was interested in the link”, “Only if strictly necessary” and

“Otherwise”]

For answering this question, participants had to imagine the links on each device

in a real webpage. The difference between the possible answers is as follows: if

the user is interested in further information but is not strictly necessary to read

such information (e.g. because it is only curiosity or because the information can

be read in another site), the participant selected the option “If I was interested in

the link”. However, if the user needs to read such information (e.g. because the

information is needed for some important reason and only can be reached from

this webpage), he/she selected the option “Only if strictly necessary”.

Q2. Would you like to be previously informed when the target of a link opens in a

new window when clicking on the link? [“Never”, “Always” and “Otherwise”]

Q3. How would you prefer to be informed when the target of a link opens in a new

window when clicking on the link? [“By an icon”, “By text” and “Otherwise”]

Questionnaires were answered after the experiment, once for each of the six devices that were included in the experiment. An additional open question (Q4) was also included:

“what is your opinion about being previously informed when a link opens its target in a new window on a mobile device?”. The first three questions were quantitatively measured using the predefined answers, and the fourth question was qualitatively analyzed by classifying similar answers. To classify similar answers, all the answers of

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. the participants were read, and those that indicated the same thing (but, of course, with different words) were put together in a group.

4.1.2.3. Results

In most cases, the task completion rate was higher when informing about the target, except in the “Same-same” case, where there was no difference (Table 5).

[Table 5 near here]

There was no statistical significant difference in “Same-same” (all cases with the same

2 2 value), “New-same” (χ2 =0.64, p=0.72) and “New-new” (χ2=1.068, p=0.586) cases, but the statistical analysis (Chi-square test) indicates that there is statistical significant

2 difference (p<0.05) in the “Same-new” case (χ3=7.891, p=0.048). This suggests that marking the target of hyperlinks impacts positively on task completion if users open links in the same window when they open by default in a new one.

The number of errors was lower when informing about the target in all cases except for the “New-new” case (Table 6), but this difference was not significant (z=-1.76, p=0.08), according to the Mann-Whitney test. There was also no statistical significant difference in “Same-same” because there were no errors in any case. However, statistical significant difference was found for the cases “New-same” (z=-2.07, p=0.04) and

“Same-new” (z=-5.11, p=0.00). Therefore, it seems that it is advisable to inform about the target of links when users open windows in a different target than the default in order to minimize the number of errors.

[Table 6 near here]

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. The average time for opening a hyperlink in the requested destination was always lower when informing about the target (Table 7), but no statistical significant difference was found in the “Same-same” case (z=-0.41, p=0.68) using the Mann-Whitney test. In the other cases, there was statistical significant difference: in “New-same” (z=-2.25, p=0.02), “Same-new” (z=-4.07, p=0.00) and “New-new” (z=-3.09, p=0.00). Therefore, informing about the target of the link is advisable in the last three cases, as it reduces the task completion time.

[Table 7 near here]

An analysis comparing effectiveness and efficiency of notifying the target with icons and text was also performed for the cases opening links in new windows, i.e., “Same- new” and “New-new”. The results (using Chi-square test) suggest that using icons instead of text is preferable for achieving the task completion in the case “Same-new”

2 (χ1 =7.894, p=0.005). In the “New-new” case, there was no statistical significant

2 difference between text and icon (χ1 =0.04, p=0.841), but the task completion percentage was higher when the notice was with an icon (83.07%) than with text

(81.63%).

As seen in Table 8, in both cases (“Same-new” and “New-new”) the number of errors was also lower when using icons than when using text. However, the statistical analysis

(Mann-Whitney test) suggests that there is statistical significant difference between using text or icons when links open a new window (z=-2.50, p=0.0125), but not when opening them in the same window (z=-1.28, p=0.2034).

[Table 8 near here]

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. Finally, according to the Mann-Whitney test, no statistical significant difference was found in the task completion time between using text or icons (Table 9), nor in the

“Same-new” case (z=-0.50, p=0.6178) or in the “New-new” case (z=-0.62, p=0.5389).

[Table 9 near here]

Regarding the satisfaction, according to the answers to Q1, 64.91% of users would click on a hyperlink that by default opens a new window if they were interested in the link, while 30.70% would do it only if strictly necessary. For Q2, 73.69% of users prefer to be always previously informed when a link will open a new window on a mobile device, while only 15.79% prefer to never be informed about it. According to the results for Q3, all the users (100%) preferred to be informed by an icon in all mobile operating systems except Symbian, where 84.21% preferred to be informed by an icon and

15.79% otherwise but not by text (they said that no new windows could be opened in this device, so they considered that the way to be informed did not matter). In average

(for all operating systems together), 97.37% of users preferred to be informed by an icon and 2.63% otherwise (neither icons nor text). Finally, Q4 showed that 47.37% of users prefer to be informed when a link opens its target in a new window on mobile devices, because not all devices clearly show the number of currently open windows and this may be confusing for users, since opening links in a new window is not the default behavior of web browsers, or even because users may not want to open more windows. Furthermore, 15.79% of users answered that sometimes they do not know where they are or the number of windows that are already opened in the browser. On the other hand, 10.53% of participants answered that different operating systems manage windows in different ways: some in a comfortable way and others do not notify where the windows are opening. Another 10.53% indicated that notification of the targets of

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. links should be done in the least intrusive possible way (such as a small icon), that do not distract them when reading or browsing the page. Moreover, 5.26% of users warned that inexperienced users may open windows in some devices and then they would not able to close them; and 5.26% said that it is necessary to notify the user when opening new windows/tabs because in mobile devices the tabs are not as visible and accessible as in a PC browser, so opening new unexpected tabs may be annoying. Finally, 5.26% of users pointed out that informing about the number of windows/tabs opened as well as providing a short animation (such as the iPhone’s) would facilitate understanding the context and avoiding the user to get lost.

Results showed that task completion was higher when informing about the target of the link, having significant difference when a link that by default opens in a new window is opened in the same window. The number of errors and/or efficiency when a link opens in a new window (either opening it in the same window or in a new one) is also statistically different, so we could say that in mobile devices it is preferable to inform when a link opens by default in a new window. Results of the satisfaction survey also support this. Answers to Q2 suggest that users prefer to be always previously informed when they are using a mobile device and a link opens by default in a new window.

Statistical significant difference was also found between using icons or text to inform about the target of a link in task completion success (case “Same-new”) and in the number of errors (case “New-new”), being better for icons in both cases. Therefore, it would be advisable to use an icon instead of a text in mobile devices to inform about the target of links when they open a new window. It is important to highlight that the icon used should be easy-to-understand.

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. The conclusions for this experiment are that most current mobile devices support multi- window, and links that open new browser windows or pop-up windows should be clearly marked on mobile devices. Therefore, we can confirm the initial hypothesis: “if a link opens a new browser window or a pop-up window, it should be clearly marked” in mobile devices.

4.2. Experiment 2: Link length

Guideline 9.4.13 of ISO 9241-151 states that “textual link names should be long enough to be understood but short enough to avoid wrapping”. The purpose of hyperlinks is clicking, but before a user can click on it, she needs to be able to identify it and know where it will lead to. Hyperlinks could not be correctly identified when wrapping occurs, and that is what guideline 9.4.13 states, so the task selected to check the validity of this guideline in mobile devices was counting links, which is a way to know if the subjects were able or not to clearly identify them. A different question is being able to distinguish adjacent links, which requires some visual indicators to help users perceive the boundaries, but this is not included in this guideline but in guideline 9.4.3, which has been found to be also valid in mobile devices by the advisors of this study. To check whether this guideline is valid or not for mobile web sites, an experiment with users was designed, where the hypothesis was the guideline itself.

4.2.1.1. Test setting

Nine different interfaces of web pages containing some words or sentences marked as links were designed for this experiment: three with short links, three with medium-sized links and three with long links. These interfaces were randomized and each subject only performed the experiment with one randomized interface of each type, i.e., there were

(Figure 3(b)) and with long links (Figure 3(c)). The number of words in the text was the same for each interface. The links were always colored and underlined to maximize the perceived affordance of clickability [42], and the text used was extracted from a chapter of The Theory of Evolution by Charles Darwin. Short links were just a single word; while medium-sized links were at least two words but without taking up more than two lines; and long links occupied several lines.

[Figure 3 near here]

4.2.1.2. Methodology

The experiment compared the usability of the three types of interfaces, based on effectiveness, efficiency and satisfaction. To do this, the users had to count the number of links on each type of interface. Users had then to detect or discover the hyperlinks without opening them, to detect possible confusion when wrapping occurred. It is important to note that the links were disabled to prevent the user from finding where the link starts and ends by clicking on it. The effectiveness was measured taking into account the number of errors, using formula (1), where n is the number of real links and i is the number of links the user counted.

|푛−푖| 푒푓푓푒푐푡푖푣푒푛푒푠푠 = 1 − (1) 푛

This formula was obtained by logic: the maximum effectiveness is 1 (100%). So the effectiveness measured would be “1-x”, where “x” is the percentage of fails. To get the percentage of fails, firstly the difference between the number of real links and the number of links the user counted was obtained, but in absolute value (numerator),

(numerator) was divided by the total of real links (denominator) in order to get the percentage of fails.

Efficiency was measured by the time taken to count the number of links on each type of interface. Since efficiency may be influenced by vertical scrolling (more scrolling requires more time), which is necessary to read the text and find the hyperlinks, the number of words in the text was the same on each interface. Satisfaction was studied by collecting the answers of a questionnaire that users filled out after the experiment.

Treatments were randomly distributed among users to avoid the order effect, so that each user performed the experiment with a web page with short links, another one with medium size links and finally one with long size links, but not all subjects did it in the same order. Additionally, another web page was created with long-sized links but with justified text instead of left-aligned text. This is because a pilot test was conducted with a test subject and we realized that the justification (or not) of text could confuse the user when counting the links, especially when they take up more than one line (line wrap), as in the case of long size links.

The experiment was performed with twenty-two users, of which seventeen were male and five female. None of them participated as advisors in stage 1. Fourteen of the subjects were between 18 and 24 years; five were between 25 and 34 years; and three were older than 35 years. Of these, ten were considered experts in the use of mobile devices, ten had an intermediate level, and two were considered novices.

In total, eighty-eight videos were recorded, four for each user: (1) one with unjustified text and short size links, (2) another with unjustified text and medium size links, (3)

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. another one with unjustified text and long size links, and (4) another one with justified text and long size links.

The users filled out a satisfaction survey after the experiment, which consisted of three questions for each type of link length:

Q1. Is it comfortable to read the text?

Q2. Is the target of the link easy to understand?

Q3. What do you think, in general, you will get if you click on the links?

4.2.1.3. Results

Effectiveness was higher with short and medium links than with long links (Table 10) but, according to the Friedman test, there was no statistical significant difference

2 (χ2=3.50, p=0.174), as well as there was no statistical significant difference between justified and unjustified text (z=-0.78, p=0.440).

[Table 10 near here]

Regarding efficiency, the shorter the links the fewer the time taken to complete the task

(Table 11). The Friedman test suggests that there was statistical significant difference

2 between short, medium and long size links in terms of efficiency (χ2=20.46, p=0.00), so a pair-wise comparison was performed using the Wilcoxon test. The analysis showed that there is statistical significant difference in time between short links and long links

(z=-3.96, p=0.00), as well as between medium links and long links (z=-3.33, p=0.00).

No statistical significant difference was found between short and medium sized links

(z=-1.70, p=0.09) or between long links with justified and unjustified text (z=-0.77, p=0.44).

Regarding the satisfaction survey, the answers to Q1 and Q2 were measured quantitatively on a Likert scale from 0 to 5 (0 – totally disagree, 5 – totally agree). The basic statistics of results for each configuration are shown in Table 12. The analysis of answers to Q1 showed that there was statistical significant difference in comfort to read

2 the text depending on the size of the links (χ2=35.08, p=0.00), so a pair-wise comparison was carried out. The pair-wise comparison showed that there was statistical significant difference in comfort to read between all sizes of links, i.e., between short and medium sized links (z=-3.30, p=0.00), between short and long links (z=-4.04, p=0.00) and between medium and long sized links (z=-3.87, p=0.00). Therefore, we could say that the shorter the links the more comfort for users to read the text. However, there was no statistical significant difference between justified and unjustified text for long links (z=-0.04, p=0.97).

[Table 12 near here]

The analysis of answers to Q2 showed that there was no statistical significant difference

2 between the size of links regarding the ease of understanding their target (χ2=5.55, p=0.06). No difference between justified and unjustified text was found regarding the ease of understanding the target of links (z=-0.83, p=0.41).

The answers of the subjects were free for Q3, i.e., there were no predefined answers and they could answer freely what they wanted. Many of the users (90.91%) thought that short links would lead to a web page where the definition of the word was explained, while long and medium sized links usually made users to think (77.27% of cases) that the target was a web page that explained in more detail the link sentence.

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. Long links were the lowest rated item for Q1 and Q2, whereas the highest mean is for short links. The mean of medium size links is between the short and long links. This suggests that users prefer links as short as possible in mobile web sites, as well as the target of links is better understood when they are shorter. Efficiency was better for the shorter links, so the initial hypothesis (“Texts of the links should be long enough to be understood and short enough to avoid wrapping”) does not hold in the case of mobile devices, because users seem to prefer links as short as possible, and also the data suggest that short and medium links are identified in a faster way than long links.

Therefore, the guideline should be modified and stated as follows for mobile web sites:

“Texts of the links should be as short as possible to be understood”.

5. Proposal of guidelines

The last stage is to propose the final guidelines about hyperlinks for mobile devices. To do this, the guidelines obtained after stages 1 and 2 were put together, i.e., from stage 1, the guidelines whose answers did not follow a random distribution and that were mainly classified as “b)” (directly applicable); and from stage 2, the guidelines consistent with the results of the experimentation.

Therefore, the final set of usability guidelines about the design of hyperlinks for mobile web sites is as follows:

1. Identification of links. Links should be easily recognizable by the user in mobile

devices.

2. Distinguishing adjacent links from each other. When several textual links are

shown in one section of text or in a single line in mobile devices, the links

should be visually separated from one another.

a page should be chosen so that mobile users can easily distinguish between

navigation and transactions manipulating data.

4. Self-explanatory link cues. Link cues (e.g. link labels, icons or tool-tips)

presented to the user in mobile devices should be self-explanatory and give a

clear indication of the target to which the link leads.

5. Using familiar terminology for navigation links. Mobile navigation links should

be labelled with terms that are familiar to the user.

6. Using descriptive link labels. In mobile devices, the target or purpose of a link

should be directly indicated by its label, avoiding generic labels.

7. Highlighting previously visited links. Links that have been previously visited by

the user should be marked in mobile devices.

8. Marking links to special targets. Links leading to special targets in mobile web

should be clearly marked with an appropriate indication of the special target

characteristics.

9. Marking links opening new windows. Links that open new browser windows or

pop-up windows in mobile devices should be clearly marked.

10. Distinguishing navigation links from controls. Navigation links in mobile

devices should be clearly distinguishable from controls activating some action.

11. Distinguishable within page links. Within-page links in mobile web should be

clearly distinguishable from other links that lead to a different page.

12. Link length. The texts of hyperlinks in web sites for mobile devices should be as

short as possible to be understood.

13. Redundant links. The labels of the redundant links should be consistent in

mobile web sites.

be formatted so that the presence of links does not impede the readability of the

text in mobile devices.

15. Page titles as bookmarks. Pages should have appropriate titles in mobile web

sites.

To summarize, initially there were fifteen guidelines for PC, from which thirteen were directly kept for mobile web sites after the evaluation by advisors, and two underwent experimentation. After experimentation, one guideline was kept in its current form and the other one was modified to be adapted for mobile web sites. Fourteen were maintained for mobile devices like they were stated for PC and one was modified.

Therefore, we could argue that most of traditional (PC) navigation guidelines can be applied to mobile devices, but it cannot be assumed a priori that every guideline for navigation in PC is valid for mobile devices.

It is important to note that these guidelines would be valid for mobile web interfaces on touchscreen smartphones, since this study was carried out in this context.

6. Discussion

It can be observed that only two of the guidelines proposed already existed in other standards and recommendations for mobile devices. More specifically, guideline 9.4.5

(using self-explanatory link cues) has already been stated by Budiu and Nielsen [36] and W3C [34], as well as it was also defined for PC by the US Department of Health and Human Services (DHHS) [32]; and guideline 9.4.6 (using familiar terminology for navigation links) was already stated by W3C [34] for mobile devices, and also for PC by the DHHS [32].

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. On the other hand, the rest of navigation guidelines proposed in this paper have not been previously proposed for mobile devices in any standard or recommendation, however, two of them have been stated for PCs. It is the case of guidelines 9.4.7 (using descriptive link labels) and 9.4.8 (highlighting previously visited links), which have been proposed for PC web sites by DHHS [32] in its guidelines 10.1 and 10.7 respectively. Guideline 9.4.7 has also been stated by W3C in WCAG [35], but also for

PC web sites.

This paper proposes eleven guidelines for mobile web sites that cannot be found in previous standards or recommendations. Ten of them (9.4.2 to 9.4.4, 9.4.9 to 9.4.12 and

9.4.14 to 9.4.16) have been stated by ISO [33] for PC and we firstly checked whether they were valid for mobile devices or not, and one is completely new. The new guideline came out from the experimentation for guideline 9.4.13, and new results suggest changing the traditional guideline for PC to a guideline adapted for mobile devices. The difference between the traditional guideline and the guideline for mobile web sites is that in PC link names should be long enough to be understood but short enough to avoid wrapping, while in mobile devices link names should be as short as possible to be understood. The fact that some of the proposed guidelines were stated for

PC could not directly imply that they are valid for mobile devices because traditional usability guidelines may not be valid for mobile devices [54], so experimentation was needed to check it.

One of the limitations of this study is that the profiles of the advisors as well as the subjects of experiments are not heterogeneous and highly representative of general users of mobile devices. However, it is difficult to find advisors and participants who are

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. willing to participate in experiments, so sometimes it is very complicated to get a representative sample.

Additionally, only Section 9 of ISO 9241-151 was considered for this study. This does not include two guidelines about hyperlinks that were identified during the literature review, so a similar method could be used to check their validity for mobile devices or not.

It should also be taken into account that there is no evidence that the icon used in

Experiment 1 is usable. None of the participants asked about what the icon meant, so we assumed that the icon was easily understood, but this should be checked.

Furthermore, semantics were not empirically measured in Experiment 2 (i.e., we focused on link size and not on understanding of links), although we considered it as a subjective matter in the survey. Another limitation is that Experiment 2 was performed only with one device, so it could have been studied if there are differences in the results when changing the interaction technique (e.g. keyboard instead of touch screen) and the screen size (e.g. using tablets).

On the other hand, Nielsen [64] showed that twenty users are enough for getting valid results in experiments using quantitative measures. However, the experiments with users as well as the initial evaluation with advisors did not always reach that number, so more participants would be necessary to get more data and make statistical analysis more robust.

In the second experiment, justification of text was only tested with long links. As a future work, it would be interesting to extend the second experiment to check also if justified text may confuse the user when counting short and medium links.

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. Finally, in order to extend our study to other devices such as tablets (i.e. to know if the guidelines proposed are also valid for those devices), the experiments should be repeated using this type of devices, because only smartphones were used in this study.

7. Conclusions

Due to the different characteristics of mobile devices from those of PCs, it cannot be directly assumed that traditional usability guidelines are valid for mobile devices. This paper shows a complete process for checking and adapting usability guidelines of hyperlinks for PCs to mobile devices. Fifteen guidelines have been stated for hyperlinks in mobile devices, from which ten are new because they did not exist in mobile standards or recommendations and one is completely new because it has not been stated in any standard or recommendation.

We conclude that most of current mobile devices are able to manage multiple browser windows. This contrasts with W3C indications [34] several years ago (in 2008), and is probably because mobile devices have evolved in recent years by adding new functionalities, and these advances in mobile technology have not been considered in any new standard. The guidelines proposed in this paper are expected to be valid for most of touchscreen mobile devices. Furthermore, we also conclude that it is preferable to inform the user when the target of a link opens in a new window in mobile web sites, since it is more effective and efficient, and users prefer to be always previously informed in those cases. It was also found that icons instead of text should be used in mobile devices to notify it, and they should be easy to understand.

Results also suggest that mobile web users prefer links as short as possible. They also understand better the target of links the shorter they are. The efficiency was also better

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. the shorter the links, so it is recommended to use links as short as possible to be understood in mobile web sites, in order to achieve the best efficiency and users’ satisfaction.

The guidelines proposed in this paper are up to date for designing hyperlinks in touchscreen mobile devices (smartphones), so they will be very helpful for web designers, who will improve the usability of their designed websites by applying these guidelines.

8. Acknowledgements

This work was supported by the Spanish Government (Ministerio de Educación, Cultura y Deporte) [grant number AP2010-0499].

9. References

[1] A. Penev and R. K. Wong, "Grouping hyperlinks for improved voice/mobile accessibility," in 2008 international cross-disciplinary conference on Web accessibility (W4A), 2008, pp. 50-53.

[2] D. A. Cook and D. M. Dupras, "A Practical Guide To Developing Effective Web-based Learning," Journal of General Internal Medicine, vol. 19, pp. 698-707, 2004.

[3] W. Lee, et al., "Mobile web navigation in digital ecosystems using rooted directed trees," IEEE Transactions on Industrial Electronics, vol. 58, pp. 2154-2162, 2011.

[4] W.-K. Chang and M.-H. Chuang, "Validating hyperlinks by the mobile-agent approach," Tunghai Science, vol. 3, pp. 97-112, 2001.

[5] A. Kaikkonen, "Usability Problems in Today's Mobile Internet Portals," in 2nd International Conference on Mobile Technology, Applications and Systems, 2005, pp. 1-7.

[6] A. Kaikkonen and V. Roto, "Navigating in a mobile XHTML application," in SIGCHI conference on Human factors in computing systems, 2003, pp. 329-336.

[7] A. Kaikkonen and V. Roto, "XHTML in mobile application development," in International Conference on Mobile Human-Computer Interaction, 2002, pp. 344-348.

[8] B. Ballard and B. Miller, "WML Style Guide for the Phone. com 4. X Browser™," Sprint PCS White Paper, March, vol. (Cited by: Kaikkonen, A., 2005. Usability

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. Problems in Today's Mobile Internet Portals, Mobile Technology, Applications and Systems, 2005 2nd International Conference on. IEEE, pp. 1-7.), 2001.

[9] M. Firtman, Programming the mobile web. Sebastopol: O'Reilly Media, Inc., 2013.

[10] C.-C. Hsu, "Factors affecting webpage’s visual interface design and style," Procedia Computer Science, vol. 3, pp. 1315-1320, 2011.

[11] H. Weinreich, et al., "HyperScout: Darstellung erweiterter Typinformationen im World Wide Web—Konzepte und Auswirkungen," in Mensch & Computer 2003, ed: Springer, 2003, pp. 155-164. (Cited by: Meng, L., Reichenbacher, T., 2005. Map-based mobile services. Springer.).

[12] Y. Nakamoto, "Hyrax - A Hyperlink-Based Application Framework for Smart Devices," in Third IEEE Workshop on Software Technologies for Future Embedded and Ubiquitous Systems (SEUS 2005), 2005, pp. 58-62.

[13] S. S. Chan, et al., "Usability for Mobile Commerce Across Multiple Form Factors," Journal of Electronic Commerce Research, vol. 3, pp. 187-199, 2002.

[14] A. Geven, et al., "Depth and breadth away from the desktop: the optimal information hierarchy for mobile use," in Proceedings of the 8th conference on Human-computer interaction with mobile devices and services, 2006, pp. 157-164.

[15] V. Hollink, et al., "Navigation behavior models for link structure optimization," User Modeling and User-Adapted Interaction, vol. 17, pp. 339-377, 2007.

[16] S. Shrestha, "Mobile web browsing: usability study," in 4th International Conference on Mobile Technology, Applications, and Systems and the 1st International Symposium on Computer Human Interaction in Mobile Technology (Mobility '07), Singapore, 2007, pp. 187-194.

[17] M. Vigo and S. Harper, "Coping tactics employed by visually disabled users on the web," International Journal of Human-Computer Studies, vol. 71, pp. 1013-1025, 2013.

[18] T. Westerveld, et al., "Retrieving web pages using content, links, urls and anchors," in Tenth Text REtrieval Conference (TREC 2001), Gaithersburg, 2002.

[19] C. B. Mills and L. J. Weldon, "Reading text from computer screens," ACM Computing Surveys (CSUR), vol. 19, pp. 329-357, 1987.

[20] M. C. Dyson and G. J. Kipping, "The effects of line length and method of movement on patterns of reading from screen," Visible Language, vol. 32, p. 150, 1998.

[21] M. Böcker, "A multiple index approach for the evaluation of pictograms and icons," Computer Standards & Interfaces, vol. 18, pp. 107-115, 1996.

[22] K. Hinckley, et al., "Quantitative analysis of scrolling techniques," in Proceedings of the SIGCHI conference on Human factors in computing systems, 2002, pp. 65-72.

[23] M. Jones, et al., "Improving Web interaction on small displays," Computer Networks, vol. 31, pp. 1129-1137, 1999.

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. [24] P. Eslambolchilar and R. Murray-Smith, "Tilt-based automatic zooming and scaling in mobile devices–a state-space implementation," in International Conference on Mobile Human-Computer Interaction, 2004, pp. 120-131.

[25] M. Maguire and M. Tang, "Comparison test of website use with mobile phone and laptop computer," in International Conference on Human-Computer Interaction, 2014, pp. 146-154.

[26] A. Parush and N. Yuviler-Gavish, "Web navigation structures in cellular phones: the depth/breadth trade-off issue," International Journal of Human-Computer Studies, vol. 60, pp. 753-770, 2004.

[27] C.-C. Lai and C.-F. Wu, "Display and device size effects on the usability of mini- notebooks (netbooks)/ultraportables as small form-factor Mobile PCs," Applied ergonomics, vol. 45, pp. 1106-1115, 2014.

[28] A. Dillon, et al., "The effects of display size and text splitting on reading lengthy text from screen," Behaviour & Information Technology, vol. 9, pp. 215-227, 1990.

[29] ISO, "ISO 9241-11: Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs): Part 11: Guidance on Usability," ed, 1998.

[30] E. Garcia-Lopez, et al., "An experiment with content distribution methods in touchscreen mobile devices," Applied ergonomics, vol. 50, pp. 79-86, 2015.

[31] E. Garcia-Lopez, et al., "Comparing zooming methods in mobile devices: Effectiveness, efficiency and user satisfaction in touch and non-touch smartphones," International Journal of Human-Computer Interaction, vol. 31, pp. 777-789, 2015.

[32] DHHS. (2011, 2015-11-25). Usability.gov: Improving the user experience. Available: http://www.usability.gov

[33] ISO, "ISO 9241-151: Ergonomics of human-system interaction -- Part 151: Guidance on World Wide Web user interfaces," ed, 2008.

[34] W3C, "Mobile Web Best Practices 1.0: basic guidelines," ed, 2008.

[35] W3C, "Web content accessibility guidelines (WCAG) 2.0," ed, 2008.

[36] R. Budiu and J. Nielsen, Usability of Mobile Websites and Applications: Design Guidelines for Improving the User Experience of Mobile Sites and Apps. Fremont: Nielsen Norman Group, 2011.

[37] C. H. Leung, et al., "Analysis of mobile commerce market in Hong Kong," in 5th international conference on Electronic commerce, 2003, pp. 408-412.

[38] V. Hollink, et al., "A semi-automatic usage-based method for improving hyperlink descriptions in menus," International Journal of Human-Computer Studies, vol. 67, pp. 366-381, 2009.

[39] J. Sprengers, "Objective evaluation of likely usability hazards–preliminaries for user testing," 1999.

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. [40] J. Nielsen. (2010, 2015-11-25). 113 Design guidelines for homepage usability. Available: http://www.nngroup.com/articles/113-design-guidelines-homepage-usability

[41] R. Bos, et al., "Heuristic evaluation of content management systems: CMS specific heuristics," in IADIS International Conference, 2005, pp. 247-254.

[42] J. Nielsen. (2004, 2015-11-25). Guidelines for visualizing links. Available: http://www.nngroup.com/articles/guidelines-for-visualizing-links

[43] J. Ling and P. Van Schaik, "The effects of link format and screen location on visual search of web pages," Ergonomics, vol. 47, pp. 907-921, 2004.

[44] L. Wroblewski and E. M. Rantanen, "Design considerations for web-based applications," in Human Factors and Ergonomics Society Annual Meeting, 2001, pp. 1191-1195.

[45] P. Van Schaik and J. Ling, "The effect of link colour on information retrieval in educational intranet use," Computers in Human Behavior, vol. 19, pp. 553-564, 2003.

[46] I. De Ridder, "Visible or invisible links: Does the highlighting of hyperlinks affect incidental vocabulary learning, text comprehension, and the reading process," Language Learning & Technology, vol. 6, pp. 123-146, 2002.

[47] M. Grahame, et al., "Age differences in search of Web pages: The effects of link size, link number, and clutter," Human Factors: The Journal of the Human Factors and Ergonomics Society, vol. 46, pp. 385-398, 2004.

[48] M. Ziefle, et al., "How younger and older adults master the usage of hyperlinks in small screen devices," in SIGCHI conference on Human factors in computing systems, 2007, pp. 307-316.

[49] P.-L. P. Rau, et al., "Implementing usability engineering in the development of a children’s community Web site," in 5th Asia Pacific Conference on Computer Human Interaction (APCHI 2002), Beijing, 2002.

[50] C.-H. Yeh and Y.-C. Lin, "User-centered design of web pages," in Computational Science and Its Applications – ICCSA 2008, O. Gervasi, et al., Eds., ed New York: Springer, 2008, pp. 129-142.

[51] Y. Cui and M. Honkala, "A novel mobile device user interface with integrated social networking services," International Journal of Human-Computer Studies, vol. 71, pp. 919-932, 2013.

[52] T. Tirapat and T. Achalakul, "Usability assessment for hyperlink methods," in International Conference on Hybrid Information Technology (ICHIT'06), 2006, pp. 252-256.

[53] M. Albers and L. Kim, "Information design for the small-screen interface: an overview of web design issues for personal digital assistants," Technical Communication, vol. 49, pp. 45-60, 2002.

[54] J. Nielsen and R. Budiu, Mobile Usability. Berkeley: Pearson Education, 2013.

[55] A. Stevenson, Oxford dictionary of English: Oxford University Press, USA, 2010.

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. [56] T. R. Williams, "Text or graphic: An information processing perspective on choosing the more effective medium," Journal of technical writing and communication, vol. 23, pp. 33-51, 1993.

[57] M. B. Curry, et al., "The effects of the visual metaphor in determining icon efficacy," in Human Factors and Ergonomics Society Annual Meeting, 1998, pp. 1590-1594.

[58] E. Garcia, et al., "Semiautomatic Evaluation of Websites Usability," in Fifth International Conference on Advances in Computer-Human Interactions (ACHI 2012), 2012, pp. 165-169.

[59] B. Rummel, "Probability plotting: a tool for analyzing task completion times," Journal of Usability Studies, vol. 9, pp. 152-172, 2014.

[60] J. Sauro. (2010, 9th January 2017). What To Do With Task Times When Users Fail A Task. Available: http://www.measuringu.com/blog/failed-times.php

[61] J. Brooke, "SUS-A quick and dirty usability scale," Usability evaluation in industry, vol. 189, pp. 4-7, 1996.

[62] J. Kirakowski and M. Corbett, "SUMI: The software usability measurement inventory," British journal of educational technology, vol. 24, pp. 210-212, 1993.

[63] K. Hornbæk, "Current practice in measuring usability: Challenges to usability studies and research," International Journal of Human-Computer Studies, vol. 64, pp. 79-102, 2006.

[64] J. Nielsen, "Quantitative studies: How many users to test," Alertbox, June, vol. 26, p. 2006, 2006.

Eva Garcia-Lopez has a BSc (2007) and a MSc (2009) in Computer Science from the

University of Alcala (Spain), where she is now an assistant professor in the Computer

Science Department. She finished her PhD (2013) in Information and Knowledge

Engineering. Her research interests include usability and mobile devices. She participates in different research projects and she has more than 60 publications in relevant conferences and journals.

Antonio Garcia-Cabot has a MSc (2009) in Computer Science from the University of

Alcala (Spain), where he now occupies an assistant professor position in the Computer

Science Department. He finished his PhD (2013) in Information and Knowledge

Engineering. His research interests include intelligent agents, adaptable and adaptive systems, mobile devices and usability. He has more than 60 publications in relevant conferences and journals. He participates in different national and European research projects.

Cristina Manresa-Yee has a PhD (2009) in Computer Science from the University of

Balearic Islands (Spain), where she now occupies an associate professor position. Her research interests include human-computer interaction (HCI), usability and vision-based interfaces. She is a member of ACM SIGCHI.

Luis de-Marcos has a BSc (2001) and an MSc (2005) in Computer Science from the

University of Alcala, where he also completed his PhD in the Information,

Documentation and Knowledge program in 2009. He is currently working in the

Computer Science Department as an assistant professor. He has over 90 publications in relevant conferences and journals. He has also been a research fellow at the Lund

Technology (Mexico).

Carmen Pages-Arevalo has a MSc in Computer Science from the Polytechnic

University of Madrid. She has a PhD from the University of Alcala, where she is currently working in the Computer Science Department as assistant professor. She has relevant publications in different conferences and journals. Her research lines are related to information systems, software quality and methodologies.

Table and Figure captions

Table 1. Comparative of the guidelines about hyperlinks found during the literature review

Table 2. Characteristics of the advisors participating in the study

Table 3. Chi-square analysis about answers in advisors’ evaluation

Table 4. Devices used in part 1 of the experiment

Table 5. Task completion percentages in Experiment 1

Table 6. Basic statistics of the number of errors when opening links in Experiment 1

Table 7. Basic statistics of the efficiency (time in seconds) when opening links in

Experiment 1

Table 8. Basic statistics of the number of errors in opening links when informing about the target with icon and text in Experiment 1

Accepted manuscript: Validation of navigation guidelines for improving usability in the mobile web. E. Garcia-Lopez, A. Garcia-Cabot, C. Manresa-Yee, L. de-Marcos, C. Pages-Arevalo. Computer Standards & Interfaces. 2017. Table 9. Basic statistics of the efficiency (time in seconds) in opening links when informing about the target with icon and text in Experiment 1

Table 10. Results obtained for effectiveness in Experiment 2

Table 11. Results obtained for efficiency in Experiment 2

Table 12. Results obtained for satisfaction in Experiment 2

Figure 1. Mechanism used to record user interactions

Figure 2. Categories of tasks performed by users in Experiment 1

Figure 3. Prototype of web pages designed for Experiment 2: (a) short, (b) medium and