The challenge of web guidelines: Investigating issues of awareness, interpretation, and efficacy

by

Stephen James Szigeti

A thesis submitted in conformity with the requirements for the degree of Doctor of Philosophy Faculty of Information University of Toronto

© Copyright by Stephen James Szigeti 2012 The challenge of guidelines: Investigating issues of awareness, interpretation, and efficacy

Stephen James Szigeti Doctor of Philosophy Faculty of Information University of Toronto 2012

Abstract

Guidelines focusing on web interface design allow for the dissemination of complex and

multidisciplinary research to communities of practice. Motivated by the desire to better

understand how research evidence can be shared with the web design community, this

dissertation investigates the role guidelines play in the design process, the attitudes

hold regarding guidelines, and whether evidence based guidelines can be consistently interpreted

by designers. Guidelines are a potential means to address the knowledge gap between research

and practice, yet we do not have a clear understanding of the relationship between research

evidence, guideline sets and web design practitioners.

In order to better understand how design guidelines are used by designers in the practice

of web interface design, four sequential studies were designed; the application of a guideline

subset to a design project by 16 students, the assessment of ten health information websites by

eight designers using a guideline subset, a web based survey of 116 designers, and interviews

with 20 designers. The studies reveal that guideline use is dependent on the perceived trustworthiness of the guideline, its source and the alignment between guideline advice and experience. The first two studies found that guidelines are inconsistently interpreted.

One third of the guidelines used in the second study were interpreted differently by participants,

ii an inconsistency which represents a critical problem in guideline use. Findings showed no difference in the characteristics of guidelines which were consistently interpreted and those for which interpretation was the most inconsistent. Further, research evidence was not a factor in guideline use, less than half the designers are aware of evidence-based guideline sets, and guidelines are predominantly used as memory aids. Ultimately alternatives to guidelines, such as checklists or pattern libraries, may yield the best results in our efforts to share research knowledge with communities of practice.

iii

Acknowledgements

While thanks are due to many people who helped me in my journey to complete this dissertation,

I reserve the greatest thanks and gratitude to my supervisor, Dr. Joan Cherry. Her guidance,

wisdom and support throughout my time in the doctoral program have shaped my understanding

of what it means to be a good researcher.

I am grateful for the excellent advice and guidance offered by my committee members

Dr. Mark Chignell, Dr. Kelly Lyons and Dr. Chun Wei Choo. Mark not only contributed to the

design of the research, but offered support and a different perspective that enriched my work. At

various key moments, Kelly and Chun Wei offered a welcome mix of reassurance and suggestions. Thanks to my external appraiser, Dr. Gary Perlman, for his insights and invaluable

comments.

I am also grateful for the involvement of Dr. Elaine Toms, under whose guidance I

started this journey, and Dr. Paul Muter, who was a great source of advice in the early stages. I

wish to thank fellow researchers who provided help, including M. Max Evans, Luanne Freud,

Glen Farrelly, Colin Furness, Rhonda McEwen, and Scott Pacquette. Thanks to Adam Fiser and

Tamás Dobozy for good advice. Thanks are due to the Inforum staff for their help, and to my

colleagues in the field of design for their support, including Peter Mansour and Petros Petrakis.

Involving the design community in my research was made easier through the help of Karel

Vredenburg and Ian Chalmers.

Thanks to my brother Dave for not only finding me a place to write in Berlin, but for his

support and good humour. Thanks to Alex, Zoë and Will who are not only glad to have their dad

back from his studies, but who can finally see that school does indeed have an end. Finally, thanks to my partner Natalie for her patience, encouragement, support and for being a keen and tireless reader during the final years of my doctoral research.

iv

Table of Contents

Abstract ...... ii Acknowledgements ...... iv List of Figures ...... viii List of Tables ...... ix List of Appendices ...... xi

Chapter 1: Introduction ...... 1 1.1 Introduction ...... 1 1.2 Research context ...... 1 1.3 Definitions...... 3 1.4 Purpose and significance of research ...... 7 1.5 Scope and delimitations ...... 8 1.6 Central research question and structure of dissertation ...... 8

Chapter 2: Literature Review ...... 10 2.1 Introduction ...... 10 2.2 Guidelines outside the domain of Human Computer Interaction ...... 10 2.3 Knowledge translation theory ...... 13 2.4 Interface Design Guidelines ...... 18 2.4.1 Guidelines supported by expert opinion ...... 21 2.4.2 Guidelines supported by research evidence ...... 32 2.5 Research regarding interface design guidelines ...... 42 2.5.1 Interpretation of guidelines ...... 43 2.5.2 Tools for the use of interface design guidelines ...... 46 2.5.3 The relationship between user performance and compliance with interface design guidelines ...... 51 2.6 Research regarding the user of guidelines: the designer ...... 55 2.7 Discussion and related research questions ...... 57

Chapter 3: The application of interface design guidelines (Study #1) ...... 61 3.1 Introduction ...... 61 3.2 Methodology ...... 64 3.2.1 Method ...... 62 3.2.2 Apparatus ...... 63 3.2.3 Participants ...... 64 3.2.4 Procedure ...... 65 3.2.5 Measures ...... 65 3.3 Results ...... 66 3.3.1 Guideline use in the design of projects ...... 66 3.3.2 Focus group sessions ...... 71 3.4 Summary ...... 82

v

Chapter 4: The interpretation and use of guidelines for web-site assessment (Study #2) .... 87 4.1 Introduction ...... 87 4.2 Methodology ...... 87 4.2.1 Method ...... 87 4.2.2 Apparatus ...... 89 4.2.3 Participants ...... 91 4.2.4 Procedure ...... 91 4.2.5 Measures ...... 91 4.3 Results ...... 93 4.3.1 Assessment of cancer health information websites ...... 93 4.3.2 Website assessments ...... 96 4.3.3 Characteristics of the 67 guidelines ...... 101 4.3.4 Participant discussion and the interpretation of guidelines ...... 106 4.4 Summary ...... 129

Chapter 5: Designer awareness, perception and use of interface design guidelines (Study #3) ...... 136 5.1 Introduction ...... 136 5.2 Methodology ...... 137 5.2.1 Method ...... 137 5.2.2 Apparatus ...... 138 5.2.3 Respondents ...... 139 5.2.4 Procedure ...... 140 5.2.5 Measures ...... 141 5.3 Results ...... 141 5.3.1 Demographic data ...... 141 5.3.2 Guideline awareness and use ...... 149 5.3.3 Attitudes towards web design guidelines ...... 153 5.3.4 Specific design trade-offs ...... 164 5.3.5 General design trade-offs ...... 168 5.4 Summary ...... 172

Chapter 6: Guideline use, design trade-offs and knowledge sharing (Study #4) ...... 175 6.1 Introduction ...... 175 6.2 Methodology ...... 175 6.2.1 Method ...... 176 6.2.2 Apparatus ...... 176 6.2.3 Participants ...... 177 6.2.4 Procedure ...... 179 6.2.5 Measures ...... 179

vi

6.3 Results ...... 180 6.3.1 Use of design guidelines ...... 180 6.3.2 Factors in choosing guidelines and/or guideline sets ...... 184 6.3.3 When guidelines are used in the design process ...... 190 6.3.4 Design trade-offs ...... 195 6.3.5 Knowledge sharing ...... 200 6.3.6 Guideline format ...... 202 6.4 Summary ...... 204

7.0 Research summary and discussion ...... 207

8.0 Contributions, limitations and future work ...... 214 8.1 Introduction ...... 214 8.2 Research answers ...... 214 8.3 Contributions ...... 216 8.4 Limitations ...... 218 8.5 Future work ...... 221 8.6 Conclusion ...... 224

References ...... 226

Appendices ...... 239

vii

List of Figures

Figure 1.1: Four principle components in a human-machine system ...... 6

Figure 2.1: The knowledge-to-action framework ...... 17

Figure 4.1: Dendrogram using Ward Linkage ...... 103

Figure 5.1: Involvement of respondents in projects of varying budgets ...... 144

Figure 5.2: Personnel employed where respondent works ...... 146

Figure 5.3: Number of web designers employed in addition to respondent ...... 147

Figure 5.4: Education of respondents ...... 148

Figure 5.5: Awareness and use of specific guideline sets ...... 150

Figure 5.6: How respondents learned about web design guideline sets ...... 151

Figure 5.7: Frequency of design trade-offs ...... 169

viii

List of Tables

Table 3.1: Application of guidelines by 16 participants in design project ...... 67

Table 3.2: Guideline usage by category ...... 68

Table 3.3: Reasons guidelines were not applied to the design ...... 69

Table 4.1: Cancer Health Information websites ranked by traffic levels (measured as total unique visitors in October 2007) ...... 90

Table 4.2: Assessments by individual participants. Sites ranked by the average number of guidelines with which they complied (maximum possible was 67) ...... 93

Table 4.3: Assessments by pairs of participants. Sites ranked by the average number of guidelines with which they complied (maximum possible was 67) ...... 94

Table 4.4: Rankings of sites by individual participant assessment, paired participant assessment and traffic levels as per comScore Media Metrix data (October 2007) ...... 95

Table 4.5: Number of disagreements within pairs of participants (prior to discussion) ...... 97

Table 4.6: Sixty-seven guidelines and the number of sites for which four pairs agreed ...... 99

Table 4.7: Characteristics of guidelines with high agreement rates ...... 104

Table 4.8: Characteristics of guidelines with no agreement ...... 105

Table 5.1: Age range of respondents ...... 142

Table 5.2: Experience of respondents as web interface designers ...... 143

Table 5.3: Frequency with which guidelines help address design problems ...... 154

Table 5.4: Themes captured in response to: A designer once said, “most web design guidelines are lacking key information needed to be effective.” Please share your opinions regarding the effectiveness of guidelines in solving web interface design problems...... 159

Table 5.5: Respondent experience with specific trade-offs ...... 165

Table 5.6: Trade-offs reported by respondents ...... 170

ix

Table 6.1: Interview participant demographics ...... 178

Table 6.2: The web interface design process ...... 191

Table 6.3: References to knowledge sharing between designers ...... 201

Table 6.4: Comments regarding guideline formats ...... 203

x

List of Appendices

Appendix A: Havelock’s seven factors which aid in understanding the phenomena of dissemination and utilization (D&U) ...... 237

Appendix B: of Bull’s-Eye Framework ...... 238

Appendix C: Questionnaire distributed to participants (Study #1) ...... 239

Appendix D: Focus group discussion script (Study #1) ...... 252

Appendix E: Guideline subset (Study #2) ...... 254

Appendix F: Assessment checklist (Study #2) ...... 256

Appendix G: Questionnaire Instrument (Study #3) ...... 260

Appendix H: Feedback form for pretest of questionnaire (Study #3) ...... 272

Appendix I: Recruitment Messages (Study #3) ...... 275

Appendix J: Interview Script (Study #4) ...... 279

Appendix K: Interviewees’ Information and Consent Form Email (Study #4) ...... 283

xi

Chapter 1

Introduction

1.1 Introduction

The principal motivations of this research are to better understand and potentially improve tools

available for web interface design. Design is an increasingly complex endeavor in which

practitioners may benefit from access to research. The means by which can be

codified and shared with communities of practice varies; but guideline sets which focus on web

interface design allow complex and multidisciplinary research to be disseminated to

practitioners. Guideline sets are a form of codified knowledge, a means to distribute research

findings and a potential design aid. A search of any major bookstore or online retailer such as

amazon.ca (2011) or indigo.ca (2011) finds myriad titles addressing the need for design help.

The role guidelines play in the design process and the attitude of practitioners regarding guidelines were unclear at the start of this research. The four studies described in this dissertation emerged from the testing of assumptions regarding guidelines and the belief that the role of guidelines needed to be better understood.

1.2 Research context

Guideline sets play a role in numerous disciplines including (Decker, 1994;

Alexander, 1977), (Chipley, Lawson & Holzheimer, 2004), design and (Herbruck & Umbach, 1997), education (Curran & Robinson, 2007; Davis, 2006) and, in particular, health care. Guidelines used in the context of the delivery of health care through clinical practice have led researchers to consider issues such as a lack of awareness of

Chapter 1 1

guidelines (Christakis & Rivara, 1998), barriers to the successful implementation of guidelines

(Grol et al., 1998) and unfavourable attitudes towards guidelines (Nuckols et al., 2007; Mosca et al., 2005). Specific to interface design, guidelines provide access to expert opinion, research- based evidence or both. From Smith and Mosier’s broad collection Guidelines for Designing

User Interface Software (1986) to the tighter focus of Research-Based Web Design & Usability

Guidelines (U.S. Department of Health and Human Services, 2006)1, guidelines represent a

means to codify both the experience of communities of practice and academic research. In the

past 20 years, literature considering interface design guidelines has reported uncertainty with how guidelines are understood and applied by those involved in the design process (Kabir, 2008;

Ivory & Megraw, 2005; Ivory, Sinha & Hearst, 2001; Vanderdonckt et al., 2000; Tetzlaff &

Schwartz, 1991). The application of guidelines was also explored in the development of tools to better facilitate the process of web design (Kim, 2010; Sutcliffe, Kurniawan & Shin 2005; Beier

& Vaughan, 2003; Henninger, 2000; Henninger, Lu & Faith, 1997; Ianella, 1995; Cohen et al.,

1994). Research has also evaluated the relationship between guideline compliance and user performance (Cherry, Muter & Szigeti, 2006; Ray, 2002; D’Angelo & Twining, 2000) and studied designers who use guidelines (Karam, O’Connell, & Perry, 2009; Connell, 2008; Salter

& Gann, 2003; Hunkleler & Vanakair, 2000; Michlewski, 2008; Carvalho & Dong, 2009; Tang

& Aleti, 2010; Ivory & Megraw, 2005; Ivory, Sinha, & Hearst, 2001; Vanderdonckt et al., 2000;

Campbell, 1996; Tetzlaff & Schwartz, 1991).

However, the role guidelines play in the design process, whether they represent a means to successfully transfer research knowledge, and the attitudes designers hold regarding guidelines, is unknown. If we continue to collect, publish and distribute guideline sets and

1 Research‐Based Web Design & Usability Guidelines has been published in two versions, the first in 2003 and the second in 2006.

Chapter 1 2

propose that they be considered helpful tools in the complex process of web interface design,

then we must better understand how they are perceived by practitioners. This dissertation was

informed by the assumption that communities of practice welcomed research evidence and that

guidelines were a means to translate such evidence into practice – an assumption which was

challenged by the research.

1.3 Definitions

The lack of standardized definitions in the field of Human Computer Interaction (HCI) requires

that terminology be clarified. The literature explored in this research makes reference almost

exclusively to design guidelines, as opposed to rules, edicts or similar terms.2 The Oxford

Canadian dictionary defines a guideline as, “a principle or criterion guiding or directing action”

(Barber, 1998). A rule is defined in the same work as, “a principle to which an action, procedure, etc. conforms or is required to conform.” The difference between a principle which guides and a requirement is vast. The former is intended to supplement or possibly inform

existing beliefs whereas the latter leaves little room for the designer’s opinions or expertise. The

literature in this area does not attempt to define rules for design, but rather considers guidance

intended to improve design and usability.

There are some differences, with regard to the origin of guideline sets, which should be

discussed when considering guidelines. As a form of codified knowledge, guidelines encapsulate

expert opinion, research evidence, or a combination of opinion and evidence. Expert opinion

refers to expertise that emerges from communities of practice in the form of experience-based

opinion. Practitioners working in web interface design make choices and trade-offs which inform

2 The grouping of guidelines, design rules, standards and even ergonomic algorithms together as examples of guidelines (Vanderdonckt, 1999) unnecessarily leads to confusion. It must be recognized that the intent of each of these collections differs.

Chapter 1 3

their work (McNeil, 2010; Ford & Wiedemann, 2008; Linderman & Fried, 2004; Weinman,

2002; Veen 2001; Nielsen, 2000). The knowledge gained from experience, in the absence of

research, is expert opinion. This is different from knowledge gained from research.

Research which informs web interface design can be traced back to the late 1920s and

early 1930s (U.S. Department of Health and Human Services, 2006)3. Guidelines which are

supported by cited research evidence should be differentiated from guidelines supported by

expert opinion. The citation of research evidence (National Cancer Institute, 2001; Koyani,

Bailey, & Nall, 2003; U.S. Department of Health and Human Services, 2006) allows the user to

review supporting evidence and to contextualize the guideline. Such research evidence differs

from expert opinion. Research, in the context of web design guidelines, is a term which requires

clarity. The authors of Research-based Web Design & Usability Guidelines (U.S. Department of

Health and Human Services, 2006) discuss the strength of research supporting various

guidelines. They do not define research, but write that for a guideline to have strong research

support, there must be evidence of “cumulative and compelling research” and “at least one

formal, rigorous study with contextual validity” (U.S. Department of Health and Human

Services, 2006, xxi). This dissertation assumes that a formal, rigorous study involves the testing of a hypothesis through experiments, quasi-experiments or surveys4.

3 Research‐based Web Design & Usability Guidelines (U.S. Department of Health and Human Services, 2006) references studies by Tinker and Paterson from 1928, 1929 and 1931 (which are no longer in print) which consider how the font, the colour of the font, the colour of the background and the length of the text line influences reading speed. These are all issues revisited by web interface designers.

4 I borrow here from Creswell, who discusses numerous strategies of inquiry and defines experiments as “true experiments, with the random assignment of subjects to treatment conditions, as well as quasi‐experiments that use nonrandomized [Keppel, 1991]. Included within quasi‐experiments are single‐subject designs,” and surveys as including “cross‐sectional and longitudinal studies using questionnaires or structured interviews for data collection, with the intent of generalizing from a sample to a population [Babbie, 1990]” (Creswell, 2003, p. 14)

Chapter 1 4

It is worth noting that guidelines can be supported by both expert opinion and research

evidence. When considering a particular guideline set, the former can be ascertained by

considering the background of the author and the relationship between the knowledge claims

made in presenting a guideline and the experience of the author. A reader can assume some

of authorial expertise. Research evidence supporting a guideline should only be recognized

where there is citation. If the guideline user cannot access the supporting research, then research

evidence support, even if declared by the author (Nielsen, 2000), cannot be validated. For the

purposes of this dissertation, the user of a guideline must, at a minimum, be able to review

appropriate supporting literature for a guideline to be considered supported by research evidence.

In discussing guidelines specific to design, the authors of the Research-based Web

Design & Usability Guidelines cautioned that guidelines do not represent rigid standards or

theory, but should rather, “prescribe practice with useful sets of DOs and DONTs” (U.S.

Department of Health and Human Services, 2006, p. iii). A useful set is one that provides advice

for designers to improve the usability of the interface. It is necessary to consider what makes an

interface usable and how to define good design. The two terms are connected in so far as a

usable interface exhibits aspects of good design. Conversely, poor design often results in poor

usability.

The International Organization of Standards (ISO) defines usability as the, “extent to

which a product can be used by specified users to achieve specified goals with effectiveness,

efficiency and satisfaction in a specified context of use” (ISO, 1998, p. 2). Effectiveness is the accuracy with which a user can complete a specific goal. Efficiency is a measure of the resources required, in relation to the effectiveness, to achieve the goal. Satisfaction is an emotional measure of the degree to which the user found the experience of achieving their goal positive and

Chapter 1 5 free from discomfort. The interaction with an interface must occur within a definable environment (both social and physical), which is the final ISO criteria for defining usability

(ISO, 1998).

Shackel (1991) posited that the usability of a system is dependent on the interaction of three components within a fourth (see Figure 1.1) and that good design is a result of solving the dynamic interaction of three components: user, tasks, and tool within the fourth component, the environment.

Figure 1.1: Four principle components in a human‐machine system (Shackel, 1991, p. 23)

Good design is closely coupled with usability, and it follows that the goal of designers would be ensuring usability, but as Norman (1988) has pointed out, this goal is unfortunately not always achieved. Designers can have numerous goals and not all of them align with ensuring the optimum usability of a system. Having recognized the variability of design goals (for example, experiential sites which focus not on usability but on the experience of discovery and interaction), this dissertation posits that “good” design, specifically, stands as a mark of usability.

Chapter 1 6

Finally, those involved in the design of web interfaces use numerous titles which change

over time. The Association (IxDA)5 lists job titles such as Interaction

Designer, User Experience Designer, User Interface Architect, and Visual Designer (The

Interaction Design Association, 2011). The description of duties for each of these positions

includes the design of interfaces which are intended to be viewed via a web-based browser

application. This dissertation uses the term web interface designer to describe practitioners

whose work encompasses (in whole or in part) the design of web based interfaces. Similarly, web

design guideline sets refer to sets intended for use by these practitioners.

1.4 Purpose and significance of the research

Good design is difficult, but guidelines may offer assistance, a potential means to positively

influence the decisions and trade-offs made by designers. The purpose of this research is to better understand the role of guidelines in the web interface design process. Guideline use by design

practitioners will be considered through studying how guidelines are interpreted by designers, by

asking designers about their awareness and use of evidence-based guidelines, and by exploring

the role of guidelines in the trade-offs designers face in the course of their work.

The results of this research will help us better understand the role of guideline sets in the design of web interfaces. We can better articulate designers’ attitudes towards guidelines, we more clearly understand how trade-offs influence the work of designers, and we appreciate what role, if any, guidelines play in addressing trade-offs. The results of the research also allow us to be more cognizant of the challenges inherent in interpreting guidelines, and to better understand

the relationship between design guidelines and the design process.

5 The IxDA web site states that, “with more than 15,000 members and over 80 local groups around the world, the IxDA network actively focuses on interaction design issues for the practitioner, no matter their level of experience.” (The Interaction Design Association, 2011)

Chapter 1 7

1.5 Scope and delimitations

While the literature review will show the existence of numerous guideline sets, the studies in this

dissertation are predominantly focused on The Research-based Web Design and Usability

Guidelines (U.S. Department of Health and Human Services, 2006). This guideline set offers the largest collection of guidelines, each with a clear indication of relevant research evidence. This set was chosen because the set arguably represents the strongest collection of guidelines as codified research evidence. The link between this guideline set and research evidence is

important, as is the process by which the guidelines were assembled (see Section 2.4.2).

1.6 Central research question and structure of the dissertation

The central, guiding question addressed in this dissertation is: How are design guidelines used by designers in the practice of web interface design? The main research question, as well as sub- questions which emerged from a review of literature in Chapter 2, are addressed across four studies, detailed in Chapters 3 to 6.

Research was conducted in multiple phases using an iterative approach. While the central, guiding research question informed all the studies, the completion of one study and the analysis of the data led to the development of a subsequent study. The specific questions addressed in each study are articulated in the appropriate chapters. The research described in this dissertation is arranged chronologically. Chapter two reviews the literature and situates the research with respect to the development of guidelines and existing research regarding guidelines and web interface design. The chapter also presents six research sub-questions which emerged from the literature review. Chapter three describes the , procedures and methods employed in the first study, before discussing the data collected and conclusions. This first study

Chapter 1 8

led to the design and execution of a second study, described in Chapter four. Chapter five

describes the design and presents findings of a web-based survey of web interface designers.

Chapter six discusses interviews with designers that were conducted following analysis of the

survey data described in the previous chapter. Chapter seven summarizes the research and

discusses implications. Finally, Chapter eight identifies research contributions, considers

limitations of the research and discusses future work.

Chapter 1 9

Chapter 2

Literature Review

2.1 Introduction

This chapter reviews research and publications from both academic and professional fields. The

focus throughout the chapter is on web design guidelines; how they are defined, the

characteristics of guidelines supported by expert opinion and those supported by research

evidence, and research regarding the use of guidelines. The chapter first considers guidelines in a

broad context, not specific to web interface design, but rather how guidelines are understood in

differing disciplines and how this understanding informs design. Specific sets of web interface

design guidelines are considered before discussing research regarding how guidelines are

interpreted and used by practitioners, studies which present and test tools intended to aid practitioners in the use of guidelines, and research looking at the efficacy of guidelines. The chapter also considers research with a focus on design practitioners and concludes with a summary discussion of the literature and six research sub-questions which emerge from the review and are addressed in the dissertation.

2.2 Guidelines outside the domain of Human Computer Interaction

The use of guidelines as a means to share knowledge is not unique to software development or web interface design. Disciplines such as architecture (Decker, 1994; Alexander, 1977), landscape architecture (Chipley, Lawson, & Holzheimer, 2004), design and applied arts (White,

2010; Herbruck & Umbach, 1997), and education (Serce et al., 2010; Curran & Robinson, 2007;

Davis, 2006) all make use of guidelines. The difference between these disciplines and web

Chapter 2 10

interface design can be found in the use of the word “guideline.” The discipline of architecture

uses the term to refer to rules or standards, while design arts and education use the term to refer

to best practices. Web interface design may also require implementing rules; there are technical

specifications that must be understood (for example, coding conventions) and design work may

require adherence to standards (such as accessibility requirements). But these are not

“guidelines” in the sense that a field such as architecture defines the term. In fields such as

architecture or education, the definition of guidelines differs significantly from how they are

defined in HCI.

The field of health care notably defines guidelines similarly to HCI, and there exist

parallels in the usage of the term between the two areas of practice. In a broad sense, research

regarding guidelines in the context of web interface design and health care considers (i) the

interpretation of guidelines, (ii) their use in practice, and (iii) their success or efficacy.

Shaneyfelt and Centor (2009) discuss not only the problems treating clinical guidelines as rules, but the validity of the guideline format itself. The authors posit that expert opinion in the field of medicine comes with enormous bias, and because the implied values and goals of the guidelines they review are not explicit, the guidelines are met with mistrust by practitioners. More specifically, Tricoci et al. (2009) looked at the use of cardiovascular practice guidelines by clinical practitioners and found an alarming preference for guidelines without a strong evidence base. A majority of guidelines were supported by expert advice, not research evidence, and the authors contend that not only must supporting research be improved, but that the supporting evidence for the guidelines must be clarified by the guideline writers. This concern about the quality and role of guidelines in health care has been explored by other researchers, including

Christakis and Rivara (1998) who found a lack of awareness by participants of guidelines

Chapter 2 11

pertinent to their practice, Grol et al. (1998) who report the perception of barriers to the

successful implementation of guidelines and Nuckols et al. (2007) and Mosca et al. (2005) who,

in separate studies, discuss the unfavourable attitudes clinicians have towards guidelines.

Using paper questionnaires, Christakis and Rivara (1998) randomly selected 600

pediatricians and questioned their awareness of four clinical practice guidelines (CPGs).

Awareness ranged from a low of 16% to a high of 66%, which led to recommendations on the

need to improve dissemination of CPGs (Christakis & Rivara, 1998). Even where awareness

rates of CPGs were higher, researchers have observed problems with implementation (Grol et al.

1998). Sixty-one practitioners were observed making over 12,800 decisions in their contact with

patients, during which the rate of guideline compliance was measured. Compliance ranged from

44% to 74% across the various guidelines. The researchers also assessed the guidelines for

whether they contained a number of attributes (for example, complexity of the guideline,

scientific evidence, specificity)1. They concluded that different attributes played different roles in

the successful implementation of guidelines. For example, guidelines were adhered to at a higher

rate, “when an explicit description of the scientific evidence was available and the evidence was

straightforward and not conflicting and that developers needed to understand these factors” (Grol

et al. 1998, p. 860). Developers of guideline sets, argued the authors, needed to understand the

various factors that influenced guideline adherence and address them in each guideline.

Mosca et al. (2005) looked specifically at high risk CPGs and found that the adoption of

the guidelines by physicians was based on their perceived need in specific clinical situations.

They recommended better education of physicians regarding CPGs. Nuckols et al. (2007) used a

1 The assessment involved four of the authors independently evaluating the guidelines and then comparing assessments. The authors decided that if three out of four assessments indicated the presence of a specific attribute in a guideline, there would be agreement. A structured meeting was then used to reach consensus on those assessments for which there was not a majority agreement.

Chapter 2 12

guideline evaluation instrument to assess a series of guidelines with the intention of first

establishing the technical quality of a set of guidelines and then testing if acceptability of

guidelines was related to the quality.2 The authors’ assumption, based on the literature, was that physicians have been slow to incorporate CPGs in their work due to either the inapplicability of a guideline to a particular situation or the uncertainty regarding evidence-based support for the guideline. The study found that even evidence-based guidelines were not incorporated into practice because of a perceived incompatibility with clinical situations. There was an inconsistency found between specific guideline recommendations and the physician’s interpretation of how they could be applied.

Guidelines represent a tool in the transfer of research based knowledge from medical journals to practitioners in a clinical context. Such is the perceived importance of ensuring that practitioners remain aware of the most current research, that a number of theoretical frameworks have been proposed to better understand and facilitate knowledge transfer. Before discussing guidelines specific to interface design, it is helpful to consider theoretical frameworks that may be relevant to design.

2.3 Knowledge translation theory

There exists no overarching theory regarding how research knowledge is put into practice.

Rather, a variety of theories and perspectives are situated in, and across, various disciplines

(Estabrooks, Thompson, Lovely, & Hofmeyer, 2006). The dissemination, diffusion, implementation, exchange or translation of knowledge (the terms vary depending on geographic

2 The Appraisal of Guidelines for Research & Evaluation (AGREE) instrument was developed under the guidance of the University of London in 2001 in order to help assess the quality of research reporting, the predicted validity of a CPG, and the likelihood that the guideline would achieve the intended outcome.

Chapter 2 13

location)3 is of particular concern in the field of health care, where research findings may impact

the practice of medicine and the health of patients. Davis et al (2003) argue that not only is there

a gulf between health care knowledge and the practice of health care, but that, “large gaps also

exist between best evidence and practice in the implementation of guidelines,” (p. 33) and

suggest that using frameworks which have emerged from knowledge translation theories may improve the implementation of evidence-based practice.

Knowledge translation theories of relevance to web interface designers are those concerned with translating research findings into design practice. The diffusion of innovation theory (Rogers, 2005), the research development dissemination utilization framework (Havelock,

1969) and the knowledge-to-action process framework (Graham et al., 2006) offer some guidance in understanding the process of sharing research knowledge with communities of practice.

The diffusion of innovation theory proposes a consideration of four facets in the dissemination of innovation; the communication channels, time, the relevant social system, and finally the innovation itself. These four facets work in concert, so that diffusion is, “the process by which an innovation is communicated through certain channels over time among the members of a social system” (Estabrooks et al. 2006, p. 29). The theory posits that an individual progresses through a series of stages (knowledge, persuasion, decision, implementation and confirmation), during which the communication of an innovation is evaluated using subjective perceptions instead of scientific evidence (Rogers, 2005). These perceptions depend on different

3In “Defining knowledge translation,” Straus (2009) posits that, “in the United Kingdom and Europe, the terms implementation science or research transfer and update are often used. In the United States, the terms dissemination and diffusion, research use, and knowledge transfer and update are often used. In Canada, the terms knowledge transfer and exchange and knowledge translation are commonly used” (p. 165).

Chapter 2 14 user needs. The theory outlines different user segments, from innovators and early adopters, to the laggards who are the last to adopt an innovation. Rogers (2005) argued that one could not move individuals from one segment to another, but rather that the innovation would be adopted once it met the particular needs of a segment. This has implications for the role of education in the dissemination of innovation, since the task is not to educate an entire community of practice, but rather those who are the innovators. The theory proposes two facets that may be relevant specifically to the dissemination of web interface ; (i) the existing values of potential adopters is a key factor in the process of adoption, and (ii) peer to peer communication is a critical channel for spreading knowledge. It is important to note that innovation, in the context of this theory, is not the same as a research finding. This implies that while the diffusion of innovation theory may inform our understanding of how research evidence is eventually adopted, the focus of the theory is much broader than a single research finding. The use of this theory to explain how guidelines may be adopted by practitioners is problematic, since guidelines are not innovations, but rather a potential means to disseminate knowledge.

Havelock et al. (1969) used Roger’s work to develop a framework addressing the growing volume of scientific research and the belief that such research should benefit society.

The framework outlines seven factors which can aid in understanding the dissemination and utilization (D&U) of information (see Appendix A). Of these factors, linkage is arguably the most important when considering web interface design. This factor is defined by the author as, “a series of two-way interaction processes which connect user systems with various resource systems including basic and applied research, development and practice. Senders and receivers

Chapter 2 15

can achieve successful linkage only if they exchange messages in two-way interaction and

continuously make the effort to simulate each other’s problem solving behaviour” (Havelock et al. 1969, Summary). This framework has potential applications when considering guideline

awareness and use in framing the relationship between individuals and the sharing of information. However, the terminology and concepts employed by Havelock appear somewhat dated, and while it is suggested that his concepts have appeared in subsequent health care models

(Estabrooks et al., 2006), a conceptual framework which integrates the roles of knowledge

creation and application may arguably hold greater value to the web interface design community.

The knowledge-to-action framework attempts to describe the translation of knowledge to

action as an iterative process, involving both the creation and application of knowledge (Graham

et al., 2006). In part a fusion of existing theories and frameworks (the authors have identified

over 60), the knowledge-to-action framework is intended to affect a means for change in practice

settings. Figure 2.1 illustrates the framework, which is comprised of an inverted pyramid with

three sections representing different “generations” of knowledge creation, surrounded by a cycle

with seven phases.

Chapter 2 16

Figure 2.1: The knowledge‐to‐action framework (Graham et al., 2006)

Graham et al. (2006) posit that the phases in the cycle are not linear, but rather iterative. One phase might reveal a shortcoming that was not previously identified, or which may lead to variations in the subsequent phase. The authors are clear on the role of the framework as a tool for advocacy, where knowledge translation includes continuing education, awareness of planned action theories and the need to turn knowledge into action.

Chapter 2 17

Guidelines are recognized by Graham et al. (2006) as a type of knowledge tool, one that operates within the broader framework they propose. While the work of Graham et al. emerged from health care, web interface design guidelines can also be considered a tool not only to codify research knowledge, but as a means to facilitate knowledge translation. However, the relationship of guidelines, particularly design guidelines, to knowledge translation theories or frameworks, remains unclear.

The challenges surrounding guideline use in the delivery of health care (which can be summarized as issues of awareness, interpretation, application to practice and efficacy) are mirrored by the challenges surrounding guideline use in the design of web interfaces. Concerns about the factors influencing guideline adherence (Hider, Burford, & Ferguson, 2009;

Vanderdonckt et al., 2000) and about compatibility issues between a specific guideline and a specific design problem (Kim, 2010; Ivory, Sinha, & Hearst, 2001) are present in the field of

HCI. Before considering the issues around guidelines, it is necessary to discuss the specific guideline sets that have been published for use by web design practitioners.

2.4 Interface design guidelines

Coupled with the emergence of the internet as a tool used for the transmission of information is the emergence of questions regarding design standardization (Smith & Mosier, 1986).

Standardization in commercial and continues to be viewed as beneficial in terms of both economics, particularly in the realm of international trade, and efficiency, by removing the need to reinvent the proverbial wheel (ISO, 2004). It was recognized early in the development of electronic interfaces that the very understanding of not only the system, but of the information itself, depended on the visual design and the constraints placed upon interaction

Chapter 2 18 with the system (Parsons, 1970). However, the goal of implementing standardization in web- based interfaces is difficult, since the interface is represented by numerous competing browsers which continue to issue updated versions in a competitive race to capture a higher percentage of a growing market place. The speed of this market-driven change has made the idea of standardization problematic (Vora, 2009; Weinman, 2002; Nielsen, 2000).

But if not standardization, then what could designers call upon for assistance?

Standardization is a tool of knowledge transfer; it is one manner in which existing knowledge may be codified and shared. The very rapid development of computer operating systems, browsers and the various multimedia applications which are intended to enrich the user experience (such as Adobe Flash or the Quicktime video player) demand that there be some manner through which concepts of good design may be articulated. It is largely for this reason that guidelines have remained of interest to those involved in the design and evaluation of web interfaces.

The first publicly available major collection of design guidelines focusing on the computer interface was published in 1986 by Smith and Mosier. Recognizing that good design was critical to successful system performance, comprehensive collections of guidelines for the design of user interface software were published in 1983 and 1986 (Smith & Aucella, 1983;

Smith & Mosier, 1986). With over 580 guidelines in the first report and over 940 in the second, the documents were intended, “to distil human wisdom into a form useful to software designers” (Smith & Mosier, 1986, p. 39). Guidelines on a wide range of very specific design issues are collected with references to both industry reports and academic publications.

Smith and Mosier’s collection is a template for all guideline sets that follow, including those recently published; (i) the guideline is reduced to a single statement, (ii) this statement is

Chapter 2 19

supported by a more detailed explanation which may include cross references and a visual aid,

and finally (iii) citations are provided.

Shneiderman (1987) adopts a similar template for presenting each guideline and uses a

combination of guidance (expert opinion) and literature reviews to help designers and evaluators of user interfaces. Grouped into eleven sections, the discussion of design issues specific to interactive systems includes suggestions for further research and an annotated bibliography of supporting material. As extensive as the book is, the broad focus (it does not focus only on web- based interfaces, for example) leads even the author to recognize the inability of guidelines to cover all the necessary design areas.

Technology advances continued to improve the options available to interface designers, a situation which led to another collection of very specific guidelines seven years later (Galitz,

1993). User-Interface Screen Design follows a similar template to Shneiderman’s (1987) earlier work; a mix of advice with supporting literature. Newer developments in interface design were incorporated into the guideline set and Galitz (1993) discusses the importance of the user’s cognitive skills in the design process. As Tullis (1997) points out, the focus of the collection is on the design of alphanumeric (as opposed to graphic) screens, particularly data entry screens.

The advice is similar to other collections in its brevity and the occasional use of images.

These three collections represent early attempts to organize existing research into a useful format. Section 2.4.1 will now consider more recent examples of guideline sets, including those supported by expert opinion (Apple, 2011; Ford & Wiedemann, 2008; IBM, 2011; Linderman &

Fried, 2004; McNeil, 2010; Nielsen, 2002, 2000; Sutcliffe, Kurniawan, & Shin, 2010; Veen

2001; WC3, 1999 & 2008; Weinman, 2002), and those with a combination of expert opinion and

evidence-based research (ISO, 1998, 1997a, 1997b; Koyani, Bailey, & Nall, 2003; National

Chapter 2 20

Cancer Institute, 2001; Shneiderman & Plaisant, 2009). A review of published research focusing on guideline sets and the use of guidelines will conclude this section’s review of interface design guidelines.

2.4.1 Guidelines supported by expert opinion

Since the publication of Galitz’s book in 1993, the need for design advice has grown exponentially. Online retailers such as amazon.ca and indigo.ca include computer related books as a searchable category, within which numerous web design books can be found (amazon.ca,

2011; indigo.ca, 2011). A search (in August, 2011) using the term web design in the book section of amazon.ca returned 14,586 results. While these books cover a wide spectrum of design related

issues, some generalizations can be drawn. A majority of the books are related to specific programming languages that are used in the presentation or delivery of content online. Numerous

titles provide guidance (usually for the novice user) to plan, design, construct (code), post and

finally launch a web site. A sample of books reveals that they are rarely evidence-based. Instead

they share the experience of designers (McNeil, 2010; Weinman, 2002; Veen 2001), consultants

(Pearrow, 2000) or design companies (Linderman & Fried, 2004).

Expert opinion is literally the opinion of expert practitioners, whose expertise is drawn

from both their communities of practice and personal experience. This experience may call on

scientific research, but it does not require such research for support. Instead the reader is asked to

trust the opinions based on the perceived expertise of the author. The status of expert is granted

by readers either through recognition of experience or trust. In the case of guidelines, expert

Chapter 2 21

opinion may have been derived from usability studies and other activities which, while not

conforming to definitions of evidence-based research, nonetheless have produced useful data.4

The items discussed in sections 2.2.1.1 and 2.2.1.2 have been selected because they are either (i) the product of authors whose work can be considered popular based on the number of editions (Weinman, 2002) or their ranking in terms of reported sales figures and citations

(McNeil, 2010; Ford & Wiedemann, 2008; Linderman & Fried, 2004; Cloninger, 2002; Nielsen,

2000)5 and in both cases are intended to serve as a representative sample of the much larger

collection of web design titles, or (ii) the product of a corporate entity recognized for its role as a

developer of tools and products in the area of web design. The former literature is considered the

work of individual experts in contrast to the latter which represents the work of a large group of experts in collaboration.

Individual design experts

Books authored by McNeil (2010), Ford and Wiedemann (2008), Weinman (2002), Veen ( 2001) and Linderman and Fried (2004) all present guidelines as a product of experience. In each case the specific guidelines put forward are supported by past professional experience. The authors liberally use examples of past projects in which the application of a specific approach resulted in various measures of success.

4 Evidence‐based research refers to studies supported by recognized research methodologies, including qualitative, quantitative and mixed methods studies. Publication in peer reviewed journals or conference proceedings is ideal, although reports and white papers which clearly outline methodology could also be considered evidence‐based research. Guidelines supported by evidence‐based research must list citations and the research must be accessible so that a reader can validate. Both the U.S. Department of Health and Human Services and the ISO draw a distinction, when discussing guidelines, between expert opinion and evidence‐based research, a distinction important to understanding the use and potential adoption of guidelines and discussed in this dissertation.

5 The NN/g (Nielsen & Norman, 2011) website claims that, “Jakob Nielsen's book, Designing Web Usability: The Practice of Simplicity is the number‐one best‐selling book ever about user interfaces, with more than a quarter million copies in print in 22 languages.”

Chapter 2 22

McNeil in The Web Designer’s Idea Book (2010) lists a series of examples of sites that are intended to inspire, but also includes advice to help guide designers. He uses his experience as a designer to reflect on design work and to offer advice. There is no research involved, just expert opinion. Similarly, in Weinman’s Designing Web Graphics (2002) the guidance is not strictly recorded in the form of guidelines, but rather in a series of subheadings which describe the aspect of the web page under discussion. The author includes tips, extensive colour and examples, a glossary and a list of additional resources, both in print and online.

Her opinions, compelling as they are, are not accompanied by references to published research.

Veen’s The Art & Science of Web Design (2001) is intended to mentor designers, offering a source of experience and advice. The notion of knowledge transfer, from one designer to another, can be found in all the work supported by expert opinion. While Veen’s advice on interface design and the use of metaphor is useful, much of the book focuses on how certain aspects of sites are designed. The manner in which tips are presented to the reader is neither instructive nor prescriptive enough to operate as guidelines and regardless of the promising title, the book does not contain “science.”

Forty guidelines provide assistance for designers in Linderman and Fried’s Defensive

Design for the Web (2004). Using existing sites as examples, the book catalogues the interaction of the authors with each site, resulting in a series of usability studies. Each guideline is followed by a brief example and a screen shot captures the image of the web site (either faulty or well designed) while the authors describe reasons why the design works or does not. Importantly, the steps can be replicated by readers.

The last collection considered (which like the others is presented as a sample of how literature supported by expert opinion is structured) is Ford and Wiedemann’s Guidelines for

Chapter 2 23

Online Success (2008) which presents guidelines covering a broader range of issues, not just

focused on design. However, what the authors describe as guidelines are a series of “Do’s and

Don’t’s” offered by the developers of sites highlighted in the book. The book represents shared experiences, not evidence-based guidance.

The focus of these books varies from addressing broad or generalized design issues

(McNeil, 2010; Ford & Wiedemann, 2008; Weinman, 2002; Veen 2001) to specific or narrowly defined problems (Linderman & Fried, 2004) but one consistent factor is the element of trust that comes into play, not via the biography of the writers (all of whom report an extensive client list), but rather through examples of good and bad design within the books. No research evidence is provided to back up claims regarding either specific guidelines, or the implicit criteria for judging the designs reproduced in the books. Rather, the authors argue for “common sense”.

However, common sense is a relative term, and not necessarily indicative of good design.

Nielsen is perhaps one of the best known names in interface usability testing and web design. He is the author of over 75 research papers (Nielsen, 2011a) on usability engineering and the guidelines within the comprehensive Research-based Web Design & Usability Guidelines document cite the work of Nielsen 65 times, more than any other author (U.S. Department of

Health and Human Services, 2006). Any discussion of web design guidelines must address his contributions which include evaluation methods (Nielsen, 1994a, 1994b), books (Designing Web

Usability published in 2000 and Homepage Usability published a year later), reports (two editions of Usability of Websites for Children published in 2010 and 2002, Email Newsletter

Usability, published by the Nielsen Norman Group in 2006), and bulletins6 (Nielsen, 2011b).

6 The earliest bulletin was issued in 1995. There have been 420 bulletins issued to date (July 23, 2011), with 14 published so far in 2011, and 24 issued in 2010 (Nielsen, 2011b).

Chapter 2 24

Similar to other authors basing their advice on expert opinion, the readers of his recent work are implicitly asked to trust his expert advice, without recourse to source material (Nielsen,

2000; Nielsen & Tahir, 2001). Although subheadings and section headings in his work suggest guidelines will be easily found, the reader has to tease the guideline from the text. The format, lacking a list of specific guidelines, results in a book difficult to evaluate in relation to other sets of guidelines.

Nielsen originally intended to publish a companion volume to his 2000 publication. The first book described the “what” of good websites, whereas the second was to be about the “how.”

This book was to have explained the methodologies used to generate evidence in support of

Designing Web Usability (Nielsen, 2000). Instead Nielsen, along with Marie Tahir, published

Homepage Usability (2001). The book focuses, as the title suggests, on the design of a site’s home page and considers fifty examples. The first section lists 113 guidelines (grouped under 26 headings) specific to home page design. Each guideline is followed by a more detailed description, but the specificity of the guidelines varies greatly. In addition, the authors employ a

Strength of Recommendation rating, adopting a three “star” scale and making recommendations based on user testing, as well as the “prevalent design decisions on other home pages” (Nielsen &

Tahir, 2001, p. 52).

The criteria for choosing the fifty home pages are based on their prominence as popular sites. The authors state that, “most sites came from top-10 lists of most-visited sites in the U.S. and other countries, as well as lists of the world’s largest companies. We also included the sites of a few prominent government agencies and some well-run small companies and non-profit institutions.” (Nielsen & Tahir, 2001, p. 55) The use of websites with which readers might be

Chapter 2 25

familiar is useful since the sites themselves may be accessed and judged as they should be – not

in print form, but as an interactive destination.

Nielsen also conducts research which is published in a series of reports available via his

company’s website. In terms of topics, these reports range from web usability to intranet design,

from considering specific audience segments to advice on user centred design methodologies.

Usability of Websites for Children: Design Guidelines Based on Usability Studies with Kids

(2010) is a representative example, in which empirically based research is described in support

of guideline claims.

Starting in 1995, Nielsen has posted a bi-weekly column on his website, www.useit.com

(Nielsen, 2011a). Running from 1200 to 2000 words, these columns offer updates on his research, samples from the published work available on his site for purchase, and various timely tips or insights.7 Many of the bulletins include guidelines and are available to anyone with a

valid e-mail address.

The total number of guidelines presented in Nielsen’s publications exceeds 400

(assuming that the subsections and titles in Designing Web Usability represent only one

guideline each). In addition, Nielsen has authored over twenty reports containing guidelines of varying relevance to web interface design. Obviously there is overlap and redundancy among the guidelines, but they still represent a considerable body of work.

7 For example, Nielsen rated the UK election campaign newsletters in April, 2010 based on the newsletters’ compliance with four related guidelines (Nielsen, 2011b).

Chapter 2 26

Corporate bodies

A corporate body is comprised of a collection of experts, and guidelines issued under a corporate

brand represent expert opinion. At the forefront of the business of interface design are numerous

Fortune 500 companies that are familiar to practitioners of design: Microsoft, Apple, Hewlett

Packard, General Electric, and IBM. Yet currently only Apple has web design guidelines, specific to their products, available to the public.

A survey of corporate web sites reveals that Microsoft discusses the importance of usability testing, both within the company and for those using their products. They also provide guidelines for developers working on products intended to operate in the Windows environment, but not specifically for web design (Microsoft, 2011). Similarly, Apple has numerous guides available for developing iPad, iPhone and iPod applications, and has made available Mac OS X

Human Interface Guidelines (2011), an online book (in PDF format) which is not intended specifically for web design. Rather, the guidelines are intended for the development of software used on the various Apple operating system platforms. Apple has also supported the publication of a collection of research articles related to interface design (Laurel, 1990). While no explicit mention is made of the rationale behind Apple’s support, it is not unusual for larger corporate entities to work closely with academic researchers.

The Hewlett Packard (HP) corporate website has an extensive database of reports related to various facets of their business. Usability studies, particularly those related to specific HP products, are available in the form of reports for free download. However, a search of these reports found no studies outlining specific guidelines related to web site design (Hewlett

Packard, 2011). The General Electric (GE) website indicates the presence of guidelines, but they

Chapter 2 27 are not available to the public and based on the tone of writing on the site these guidelines appear to be specific to the design of web sites within the GE domain (General Electric, 2011).

With over 400,000 employees and revenues of over $USD95.8 billion, IBM has grown into a major international company since its incorporation in 1911 (IBM, 2011). The company is involved in research, development and design of numerous hardware and software applications in the online , including the design of websites. As recently as 2007, web design guidelines were available to the public and could be downloaded from the IBM website. But IBM has ceased to maintain such guidelines. Instead, there are numerous blogs and podcasts within the corporate site which provides designers with advice and guidance regarding design.

The motivation for large companies to publish blogs, podcasts (and, in the past, guidelines) is not explicitly stated on corporate web sites, but these actions represent a relationship with the community at large. A case study by IBM employees delivered at a conference over a decade ago considered the purpose of presenting guidelines, “to provide useful and up-to-date information to HCI practitioners, to demonstrate that IBM cares about and is a leader in usability, provide a public forum for trying out new UI ideas and getting feedback, provide tools and guidelines for web developers” (Lisle, Dong, & Isensee, 1998, para. 4). The challenges of design are such that sharing knowledge may trump the protection or hording of in the interest of competitive advantage. Guidelines are one means to share that knowledge and publishing under a corporate logo can lend credibility to the advice offered.

The branding of a guideline set by a corporation or an institution (for example, the World Wide

Web Consortium) provides the reader with context for the advice which differentiates it from advice written by the individual expert. Corporate or institutional bodies represent communities of practice; not individual experts, but collections of experts.

Chapter 2 28

Institutional bodies (Web Content Accessibility Guidelines 1.0 and 2.0)

Published in 1999, the first version of the Web Content Accessibility Guidelines (WCAG) provided designers with fourteen guidelines intended to ensure accessibility to Web sites users with disabilities. This guideline set differed from other guideline sets in its sole focus on the specific design demands of ensuring accessibility; accessibility which may be direct, or through the use of assistive technology (Hackett, Parmanto, & Zeng, 2005).

Similar to other guideline sets, the fourteen guidelines follow the Smith and Mosier (1986) template, with a brief title, one sentence description, a paragraph offering more information and related notes, followed by checkpoints. There are a total of 65 checkpoints across the 14

guidelines, and these guideline details are organized by priority. Priority 1 checkpoint compliance

is intended to prevent lack of access for users, Priority 2 checkpoint compliance prevents

difficulties in access and finally Priority 3 checkpoints improve access to web documents. Each

checkpoint contains links for information on techniques to implement the guideline.

WCAG 2.0 was published in 2008 and includes more tools to help designers (via examples,

and descriptions of techniques for implementing guideline recommendations) and the replacement

of graduated checkpoints with success criteria – which are similarly graduated requirements, but

which include an improved ability to test conformance. The updated set reorganizes the guidelines

into four broad principles and through this restructuring, reduces the total number of guidelines to

twelve. It is worth noting that the ideas contained in the original 14 guidelines are present in the

restructured guidelines in version 2.0.

Both guideline sets were published by W3C and they are the collaborative work of experts.

However, the guidelines (including checkpoints and success criteria) do not contain references to

research. The reader is asked to accept the recommendations based in part on the expertise of the

Chapter 2 29 authors, and in part because numerous government bodies have adopted the guidelines for Web design work funded by government organizations. Compliance with WCAG 1.0 is required by government bodies such as Canada’s Human Rights Act, 1977, Australia’s Disability

Discrimination Act, the European Union’s European Parliament Resolution, 2002, the United

Kingdom’s Equality Act, 2010 and the U.S.A.’s Section 508 of the Rehabilitation Act (Rogers,

2010). Thus depending on the design context, designers may use the WCAG 1.0 and 2.0 guidelines as suggestions, or approach them as standards or required rules.

Community of experts (Pattern Libraries)

Social media has allowed for the emergence of communities of experts, where designers, connected by little more than similar experience, can share solutions via electronic channels to design problems. An online discussion of design problems within a community of practice does not constitute a guideline set, but the emergence of collections of design solutions, sometimes in the form of pattern libraries (for example), is worth considering in the context of guidelines as these knowledge repositories share some characteristics with guidelines. Pattern libraries represent another form of expert opinion. The idea of capturing design knowledge as patterns emerged from the discipline of architecture, where Christopher Alexander, in his book A Pattern

Language (1977) presented 253 design patterns in order to help architects, planners and builders make better design choices. The description of patterns, organized around themes, uses a consistent structure where a brief context is provided, then a problem statement (usually not more than a sentence in length), more extensive explanation of the problem, and finally potential solutions. Alexander argues that patterns do not exist in isolation, but rather are surrounded by and embedded in other patterns and that one cannot build in isolation. The patterns he describes

Chapter 2 30 are often abstract, as are the solutions. The intention is to allow the architect to consider and interpret the patterns and solutions and adopt them to their particular context.

Seffah (2010), in his discussion of the evolution of design patterns in HCI, recognizes

Alexander’s important role, then credits a CHI (computer-human interaction) workshop organized in 1997 as the first consideration of patterns specific to the field of HCI. Patterns have increasingly been seen as another tool to codify best practices in design (Landay & Borriello,

2003). Design patterns, which emerge from communities of practice, share some similarities with some sets of guidelines, yet they differ in their initial focus; a problem. Where guidelines may imply a problem or design issue, design patterns are more explicit. Specific patterns are organized around specific design problems, and represent a variety of solutions. Where guidelines specify a solution, design patterns offer examples of potential solutions.

Pattern libraries are collections of design problems and solutions. The content found at

Welie.com (2011), the Yahoo! library (2011), the Open Source Design Pattern library (2011) or the UI Pattern library (2011) is similarly structured. Each pattern begins with a problem statement, then solutions, guidance on when, how and why to use the solution, and finally additional examples. The libraries are available online which allows them to be fluid, emerging collections, where designers can comment on specific patterns and discuss solutions.

Similar to various guideline sets, the number of individual patterns within a pattern library can be numerous enough to make accessing a particular pattern difficult (Janeiro, Barbosa, Springer et al., 2010, 2009). As a result, research regarding pattern libraries has considered the internal and external validity of a UI pattern language (Todd, Kemp, & Phillips, 2004), software for helping designers use pattern languages (Vanderdonckt & Simarro, 2010), and tools for better organizing

Chapter 2 31

and searching for specific patterns in pattern libraries (Janeiro, Barbosa, Springer et al., 2010,

2009).

Koukouletsos et al. (2009) found that design advice in the form of patterns had a greater

impact on the performance of novice designers than did the same advice in the form of guidelines. In the study, participants were tasked with designing and evaluating a web site using an authoring tool. Performance was an evaluative measure, as judged by design experts. They found the difference in performance scores to be statistically significant, suggesting that pattern

libraries offer a method to codify knowledge better than design guidelines.

Pattern libraries, as collections of solutions to design problems, represent the expert

opinion not of an individual or a corporation, but of a group of designers. They are crowd- sourced aids which benefit from the experiences of not one expert, but an array of design experts.

What they lack, in their manner of presentation, is research evidence. If designers seek the advice of other designers, pattern libraries offer a potential solution. But if designers wish to better

understand the research informing a design solution, they must look elsewhere; to guideline sets

which are explicitly supported by research evidence.

2.4.2 Guidelines supported by research evidence

As noted in Chapter one, guidelines can be supported by expert opinion and research evidence. It has been argued that research-based guidelines, “provide web designers with the best possible information about what the relevant research seems to be “saying” about specific topics”

(Koyani & Allison, 2003, p. 696). Research evidence is recognized where there is a citation

leading to published research. The quality of research evidence is another matter, since the act of

publication does not imply validity of findings (or even findings at all, since opinion pieces and

Chapter 2 32

project outlines may be found in publications – although neither constitutes research evidence).

There are only a handful of interface design guidelines sets supported by such evidence and they will be discussed in this section.

The ISO issued a series of interface design guidelines in the late 1990s. In particular, sections of ISO 9241 will be examined with a focus on the difficulty in ascertaining research support for individual guidelines. Comprehensive collections of evidence-based guidelines were published by the National Cancer Institute of the United States (2001) and by the U.S.

Department of Health and Human Services (2003, 2006). Although Shneiderman and Plaisant’s

Designing the User Interface (2009) is structured as a text book, it also presents guidelines and advice supported by research evidence and is intended to be used by both students of design and

practitioners. The authors of these four publications searched for and considered both expert

opinion and evidence-based guidelines in an attempt to compile a single source for web interface

designers.

ISO 9241: Ergonomic requirements for office work with visual display terminals (VDTs)

The ISO standard relevant to web interface designers is ISO 9241: Ergonomic requirements for

office work with visual display terminals (VDTs). Composed of seventeen parts, this ISO standard addresses numerous issues related to design, both in terms of process and execution.8

The guidelines are intended to be useful for interface designers, the buyers or purchasers of

designed products, evaluators who wish to apply the recommendations to products, designers

8 The seventeen parts of ISO 9241 are: Part 1: General, Part 2: Guidance on task requirements, Part 3: Visual display requirements, Part 4: Keyboard requirements, Part 5: Workstation layout and postural requirements, Part 6: Environmental requirements, Part 7: Requirements for display with reflection, Part 8 : Requirements for displayed colours, Part 9: Requirements for non‐keyboard input devices, Part 10: Dialogue principles, Part 11: Guidance on usability, Part 12: Presentation of information, Part 13: User guidance, Part 14: Menu dialogues, Part 15: Command dialogues, Part 16: Direct manipulation dialogues, Part 17: Form‐filling dialogues.

Chapter 2 33

of the tools which the product designers will use and finally the end user (ISO, 1997b). For the

purposes of this dissertation, three parts were considered; ISO 9241-11, 12 and 14. The three

parts were written by the same subcommittee of Technical Committee 159 and published in

1997 and 19989. They were chosen because they are considered an important standard for

usability and have been discussed anecdotally among researchers (Jokela, Ilavari, Matero, &

Karukka, 2003) and referenced for their importance (Tripathi, Pandey, & Bharti, 2010).

ISO 9241-11 focuses not on specific design guidelines, but rather on the process of design

and usability testing. Since the focus is on the process of design, adherence to the guidelines is

difficult to assess once production has been completed. The guidelines may be useful during the planning and execution of design, but are not intended to offer designers recommendations on

design specifics.

Where ISO 9241-11 focuses on process, part 12 contains actionable guidelines. There are

100 guidelines, grouped in hierarchies, which designers can apply directly to design problems.

Criteria used to select supporting evidence are unclear. Further, the guidelines are presented

without the benefit of detailed explanation or visual examples, increasing the potential for

misinterpretation by users. A majority of the guidelines are one or two sentences in length, with

additional notes only occasionally provided.

9 The various standards are available via the ISO website, following publication by the ISO Geneva office. Strict copyright conditions limit distribution of the documents and sharing of the standards via other distribution means is prohibited. The documents are issued under a single user license agreement, and copying and sharing via electronic networks is prohibited by the terms of purchase. The separate parts of most of the ISO standards cost approximately one Canadian dollar per page, although some are priced at considerably higher levels. ISO standards are often composed of a number of parts, which may also refer to other standards (so that the cost of an entire ISO standard is dependent on the necessity to acquire complementary or related ISO documents). Very little information regarding the specific documents is available prior to purchase. Abstracts, reference lists or even a description of the documents beyond the ISO number and title are not available, making the choice of the appropriate ISO standard difficult.

Chapter 2 34

ISO 9241-14 lists guidelines specific to menu dialogues. The document includes three

appendices, an example of the application of ISO 9241-14 and a bibliography, including a cross

reference table which is intended to match the guideline with a specific source. Menu dialogues

refer to menus used in the interaction between user and computer (in the context of typical office

tasks). The 88 guidelines are grouped in a hierarchy under four main categories and various sub

categories. The recommendations cover, “menus presented by various techniques including

windowing, panels, buttons, fields, etc. These recommendations can be utilized throughout the

design process (e.g., as guidance for designers during design, as a basis for heuristic evaluation,

as guidance for usability testing)” (ISO, 1997b, p. 1).

Outside of the guidelines themselves, arguably the most useful part of this document is the

inclusion of a cross reference table. Next to a listing of guidelines are source documents. The

documents themselves are divided into three categories: research studies, guidelines and expert

opinion. Research studies account for 57 of the listed sources, guidelines account for 183, and

expert opinion for 44 (note that the same source can support more than one guideline). However,

the document does not provide the criteria used to differentiate the source material. What

constitutes expert opinion and a guideline is unclear (could a guideline not be expert opinion?), and

research studies vary greatly in quality. The 87 citations in the bibliography, along with a cross reference table, represent a level of support for the guidelines in part 14 lacking in the other two parts explored. The lack of citations related to each of the guidelines in parts 11 and 12 results in uncertainty as to the manner of research support. The ISO process implicitly suggests that readers

“trust” the technical committees and participating members, but transparency in an area such as

interface design guidelines is arguably a factor in their adoption.

Chapter 2 35

The National Cancer Institute (United States) guidelines (2001)

In the United States, the National Cancer Institute (NCI) has been responsible for spearheading the collection of evidence-based guidelines for the design of web sites. This project is noteworthy because: (i) there existed no other comprehensive list of guidelines which included the supporting research evidence and (ii) the project began the process of articulating a problem and the process of finding solutions. The problem is the lack of evidence for the large body of guidelines currently available; the solution involves focusing the efforts of the research community on areas where evidence either does not yet exist, or is not conclusive (National

Cancer Institute, 2001).

The NCI has been involved in the publication of three key documents related to interface design guidelines. The initial set of guidelines, which number 60, was published in 2001. Two years later, the U.S. Department of Health and Human Services, in concert with the NCI, published another collection of guidelines (in this case 187). In 2006 an updated collection of guidelines was published, increasing the number of guidelines to 209. While the role of a government funded health institute in the compilation and championing of such guidelines may initially appear unusual (any number of academic institutions focusing on design competencies and research could be considered a more obvious centre for work of this nature), the NCI explains its reason as follows,

Clear and effective online communication to patients, health professionals and researchers is a critical element in the overall communications strategy of the National Cancer Institute (NCI), the federal government's principal agency for cancer research. Because NCI designs and manages a significant number of cancer-related information Web sites, these guidelines were developed specifically to assist NCI Web managers, designers, and authors improve their overall design efforts and to help them base their decisions on evidence-based approaches. However, since the general principles of good design and usability are applicable to anyone who works with information Web sites, we decided to

Chapter 2 36

make our guidelines available to other federal agencies and to the general public. (National Cancer Institute, 2001, p. 1)

The NCI guidelines represent one of the first evidence-based guideline sets available and they

informed the U.S. Department of Health and Human Services guideline sets. Designers,

managers, authors or their clients and stakeholders are only part of a longer list of potential users of the NCI set, and the guidelines themselves were promoted as being applicable outside of information and search oriented websites. Concerned with the quality of health-related websites

(and access to health related information online), the NCI adopted a seven step process to compile a list of guidelines10. The project started with an identification of major sets of

guidelines, including “those available on the Web, published in books, guidelines, summary articles, etc.” (National Cancer Institute, 2001, p. 3) Then research literature was reviewed in order to identify studies which supported the guidelines, of which few were found. So few, in fact, that the NCI, “relaxed our standard and began using secondary references from textbooks, published style guides, lists of guidelines, etc. Older studies cited in our guidelines were used because they represent the best available research,” (National Cancer Institute, 2001, p. 3).

To indicate the research support for specific guidelines, the NCI developed a Strength of

Evidence scale. The five-bullet scale was further defined by three classifications of research: category A (defined as “experiments” or hypothesis testing); category B (defined as “studies” or observational evaluation and/or performance based usability tests); and category C (defined as

“observations” or expert opinion). To be allocated five bullets, a guideline must be supported by two or more category A experiments, whereas four bullets would be awarded where there was

10 The seven steps were as follows: (i) major sets of guidelines were identified; (ii) research was reviewed which validated the guidelines; (iii) gaps in the existing research were identified; (iv) new guidelines were developed with supporting research; (v) the guidelines were divided into categories; (vi) levels of evidence were established; and finally, (vii) the relative importance of each guideline was assessed in order to help designers focus on the most meaningful of the guidelines (National Cancer Institute, 2001).

Chapter 2 37

only a single category A experiment. Not one of the 60 guidelines received five bullets and only

14 received four.

The specific impact of the publication, beyond its foundational role for the subsequent

U.S. Department of Health and Human Services guidelines sets, is unknown, as there are very limited citations of the publication.11 The lack of citations might mean either that the guidelines

have had little impact on the research community (although perhaps the publication has led to an increased awareness of the individual sources cited within) or that researchers looked to a later guideline collection, related to the NCI publication. However, this remains conjecture. As mentioned, the 2001 guideline set led directly to the publication of a more comprehensive collection of guidelines two years later.

U.S. Department of Health and Human Services guidelines (2003)

Two years after the initial guidelines were published, the NCI developed a new set of guidelines which employed a similar process of development, but resulted in a very different collection. In fact, the Research-based Web Design & Usability Guidelines (Koyani, Bailey, & Nall, 2003) do not even make reference to the earlier work, although the connection between the two moves beyond the common institutional authorship. The scope of the project expanded to include more than the need for clearer health related information. The 2003 collection is intended to: (i)

“provide quantified, peer-reviewed web site design guidelines” (Koyani, Bailey, & Nall, 2003, p. xv) and (ii) “stimulate research into areas that will have the greatest influence on the creation of

usable web sites” (p. xvi).

11 Searches were conducted using Web of Science, ACM database, Scopus and Google Scholar, The Web of Science database was searched under Title, Author and Topic fields using the following terms: guidelines, NCI, National Cancer Institute, research, research based web design, and web design. The ACM database was searched using the terms guidelines, NCI, National Cancer Institute, and web design. Searches on Google Scholar and Scopus were conducted on the title “Web Design & Usability Guidelines”. Searches were conducted on four occasions over a six year period (October 2004, April 2005, November 2006 and December 2010.)

Chapter 2 38

The process for developing this guideline set involved a thorough search for existing web

design guidelines, published articles, reports and usability tests (which includes NCI technical reports and studies). A list of 500 guidelines emerged, which were deemed too unwieldy to be of use to designers. The next step was to look for similarity among guidelines which would allow them to be merged, as well as resolve guidelines which conflicted with one another or which lacked sufficient clarity in the opinion of the reviewers. This process was undertaken by the

NCI’s Communication Technologies Branch, which is purported to have expertise in “website design, usability engineering, technical communication, , computer programming and/or human-computer interaction” (Koyani, Bailey, & Nall, 2003, p. xxi). As a result, the guidelines were reduced to 398. Sixteen external reviewers then considered the relative importance of each guideline. These reviewers were split evenly between web site designers and usability specialists and were given the task of assessing how important each guideline was to the ultimate success of a web site, scoring each one using a five bullet scale from Extremely important (5 bullets) to Not important (one bullet). These rankings were used to reduce the number of guidelines to 287.

The final list of guidelines numbered 187, but the description of the methodology unfortunately stops short of explaining how 100 guidelines were discarded to reach this final number. It may have come about as a result of the next step, which was the assessment of the evidence for each guideline. Again using a five bullet scale, a Strength of Evidence rating was

applied to each guideline based on research evidence. This scale was modified from the 2001

NCI guideline set. While a five bullet scale was retained, the criteria are markedly different. The

strength of evidence required is more lenient in the 2003 publication. Whereas the criteria for the

maximum (five bullets) Strength of Evidence in the earlier publication called for two or more

Chapter 2 39

hypothesis testing studies, the 2003 publication called for only a single “formal, rigorous study”.

A four bullet ranking could be awarded where there may exist conflicting research and where a consensus “seems to be building,” which is considerably less rigorous than the earlier requirement to have at least one hypothesis testing study.

The 2003 publication does not elaborate on the easing of strength of evidence requirements. The revised criteria for evidence resulted in 14.4% of the guidelines receiving a rating of five and almost half (43.9%) receiving four or five bullets. This compares to 23.3% in the earlier publication. However, it is important to note that this increase in evidence is not all due to a shifting of criteria. The research published during or after 2001 contributed to the growing body of research. Almost one fifth (18%) of the sources cited were published after the

2001 publication of the NCI guidelines. As research continued to be focused in this area, the

evidence supporting (or contradicting) the guidelines also increased.

A total of 418 sources are referenced in the 2003 publication. The 27 guidelines with strong research support averaged 11.5 sources of evidence, with Guideline 9.4 (titled Use

Descriptive Headings Liberally) supported by the largest number of sources at 19, and Guideline

1.14 (titled Use Heuristics Cautiously) supported by six. Eighteen of the 27 guidelines are supported by research published either in 2001 or later, evidence which was not available at the time the 2001 guidelines were published.

Regarding the authors of the source documents, the 418 references are authored by 634 individuals. Of these, 64 authors are responsible for more than five references. Five authors in particular are responsible for over 55% of the cited research (Nielsen provides the highest number of

citations with 57, followed by Koyani with 47, Evans with 46, Bailey with 44 and Spyidakis with 39).

Chapter 2 40

Unlike the NCI guideline set whose impact was uncertain, numerous citations to the 2003 set appear in

Web of Science, Google Scholar and the ACM databases.

U.S. Department of Health and Human Services guidelines (2006)

The 2006 version of the U.S. Department of Health and Human Services guideline set should be

considered a second edition. While the differences between the 2001 NCI guideline set and the

2003 publication were comprehensive (as outlined above), the 2006 edition of the guidelines is

an updated version of the 2003 version that employed the same methodology. As research in the area of web interface design continued to be published, the U.S. Department of Health and

Human Services incorporated new literature to ensure the relevancy of the set.

The process of revision led to the substantial modification (updated from the 2003 edition) of 21 guidelines and the addition of 22 new guidelines. The relative importance and the strength of evidence ratings were also revised. This process was undertaken by 13 usability professionals with a reported high agreement rate level (the authors of the guideline set report

Cronbach’s alpha = .92). Although the report incorporated 49 new sources, the number of guidelines which received a rating of five dropped from 27 (in the 2003 publication) to 18. The percentage of guidelines which received a rating of four (moderate research support) or five

(strong research support) bullets dropped from 43.9% to 38.3% of the total guideline set. While the authors do not explain this change, it possibly occurred due to the inclusion of new research

(which may have contradicted or cast into doubt the strength of earlier supporting research), as well as the opinions of 13 “usability professionals” who reviewed the research and whose opinions may have differed from the individuals involved in the 2003 edition (U.S. Department of Health and Human Services, 2006, p. xxi).

Chapter 2 41

Shneiderman and Plaisant (2009)

Shneiderman and Plaisant’s Designing the User Interface (2009) is part text-book, part guideline

set and is intended for use by both instructors and practitioners. The various guidelines presented

are organized into thematic chapters (for example, User documentation and online help or

Information search). Each section ends with a Practitioner’s summary, a Researcher’s agenda and

an extensive reference list. The summary provides practitioners with tips and advice based on

issues raised within the preceding chapter, while the researcher’s agenda provides an outline of

potential research regarding the topics discussed in the chapter. The authors consider the

shortcomings of research in a particular area of interface design, and suggest specific research

projects. The reference list which concludes each chapter is extensive, useful to practitioners and

researchers alike. The tips and guidance offered in the book are supported by cited research.

2.5 Research regarding interface design guidelines

This section considers research regarding interface design guidelines in three areas; (i) research

exploring the interpretation of interface design guidelines, (ii) research describing the

development and testing of tools for the deployment of interface design guidelines, and (iii)

research regarding the relationship between user performance and compliance with interface

design guidelines.

In his review of the development of human factors design guidelines, Campbell (1996)

discusses literature exploring the use of human factors research by designers. He describes

studies where researchers interview designers of man-machine systems and automotive display

systems and find that in both cases either the published research literature viewed by the

designers or the format in which the research was presented was difficult for the designers to

Chapter 2 42 understand. If the manner in which the research findings is presented is problematic, guidelines - as a format - pose a possible solution. As Campbell points out, researchers called for improvements to the translation of findings into a format most useful to design practitioners.

Perhaps guidelines represent this useful format. As previously discussed, Guidelines for

Designing User Interface Software (Smith & Mosier, 1986) included citations attempting to link research with practice. Similarly, Shneiderman (1987) chose the format of guidelines to present research findings to design practitioners.

2.5.1 Interpretation of interface design guidelines

The authors of the guideline sets previously discussed all make reference to the process of interpreting guidelines (Galitz, 1993; U.S. Department of Health and Human Services, 2006;

ISO, 1998, 1997b; Koyani, Bailey, & Nall, 2003; Smith & Mosier, 1986; National Cancer

Institute, 2001; Shneiderman & Plaisant, 2009; Shneiderman, 1987). The advice on how best to interpret a guideline ranges from the need to translate each general guideline into a specific design rule depending on the design context (Smith & Mosier, 1986) to ensuring compliance with government accessibility regulations (U.S. Department of Health and Human Services,

2006). Even the ISO states that the guidelines presented in ISO 92141-11 are “given in the form of general principles and techniques, rather than in the form of requirements to use specific methods” (ISO, 1998, p. 1). Unlike standards, the ISO guidelines require interpretation on the part of the designer. But it is during this interpretation that problems arise.

In their study of the application of ISO draft guidelines, de Souza and Bevan (1990) asked three interface designers to use the guidelines in order to improve the design of, “a badly designed traditional menu-style interface for an electronic mail system” (1990, p. 436). The

Chapter 2 43

designers were given an opportunity to study the guideline set as well as the initial interface

before they were asked to apply the appropriate guidelines (from a set of 45). The results were

reported across three measures; the cognitive cost (which was expressed in terms of errors and

problems using the guidelines), the affective costs (which referred to feelings of distress experienced by the designers while using the guidelines) and the quality of the interface (which was expressed as the percentage of guideline violations relative to the total number of applicable guidelines). de Souza and Bevan (1990) discovered that designers either made errors or had difficulties with 91% of the guidelines. The authors analyzed the causes and made recommendations for improvements. In the case of almost every guideline they recommended

more information and clarity. With the exception of four guidelines out of 45, the guideline

information was inadequate for the designers to properly understand. The authors concluded that the ISO group “has yet to decide whether it is possible to measure conformance with a guideline standard,” and potentially favour the publication of the more rigid standards (de Souza & Bevan,

1990, p. 439).

Focusing on a different set of interface design guidelines, Tetzlaff and Schwartz (1991) found similar problems. Their study asked nine participants (two experts regarding the guidelines and seven guideline novices) to develop a paper and pencil interface design which conformed to a set of guidelines. They found four problems with the use of guidelines; (i) some guidelines were only partially understood, (ii) distinctions which were deemed critical to the guideline authors were not appreciated, (iii) some guidelines proved too difficult to understand and

implement and (iv) the guidelines ran counter to the designers’ sense of what constituted usable

design (their own expert opinion as design practitioners). The authors report that when initially

presented with the guidelines (in the form of a book), only two of the seven novice guideline

Chapter 2 44

users started by reading the introduction. The rest began the design task and looked for guideline

information only when they thought it was relevant to their work. Although interpretation of

specific guidelines appeared to be a significant problem in the opinion of the authors, they

reported that the resulting designs nonetheless largely conformed. The paper concluded not by

suggesting that guidelines might not be the ideal tool for translating research knowledge, but

rather with advice on improving the presentation of guidelines through the extensive use of

examples. The authors found that clear illustrations supporting a guideline were, “cited

spontaneously and almost universally as the preferred vehicle for learning,” by the participant

designers (Tetzlaff & Schwartz, 1991, p. 331).

Approximately three-quarters of the guidelines in Research-based Web Design &

Usability Guidelines include colour illustrations and context specific text (Koyani, Bailey, &

Nall, 2003). The examples far outweigh in number and quality those used in earlier collections,

but the use of a subset of these guidelines (48 guidelines) as a tool for assessment in a study by

Cherry, Muter and Szigeti (2006) found the level of disagreement between two evaluators to

range from 49 to 70 percent. This lack of agreement suggests that even the improvements

recommended by earlier researchers do not overcome problems perhaps inherent in the

interpretation of guidelines. Interestingly, the disagreements were easily resolved when the two

evaluators discussed how they assessed the sites, leading the authors to consider that guidelines are possibly best implemented (or the conformance of a web-based interface to guidelines best evaluated) by more than one individual. Similarly, in an earlier study by Cherry (1998) exploring the extent to which bibliographic displays in OPACs and web catalogs complied with a set of guidelines, two evaluators were able to resolve their disagreements by discussion and joint re- evaluation of the interfaces (their initial level of agreement was a relatively high 86 percent).

Chapter 2 45

Other researchers have found similar problems regarding the understanding of guidelines by designers. It has been suggested that the use of jargon or discipline-specific vocabulary is a source of problems (Vanderdonckt et al., 2000), that a lack of specificity within the guidelines makes interpretation difficult (Ivory & Megraw, 2005), and similarly, that the “high level” generalization of guidelines makes operationalizing them challenging (Ivory, Sinha, & Hearst,

2001). Hider, Burford, and Ferguson (2009), in considering the use of guidelines in Australian libraries, suggest that although documentation and guidelines are helpful, improving the overall competency and skill set of the designers was of greater importance.

Unfortunately there does not appear to be research comparing guideline formats. Format, in this context, refers to the detail used in describing each guideline, the various contexts where the guideline would be more or less applicable and the addition of illustrations and examples.

Such research might help identify the correct level of specificity, or exactly what manner of example would be most helpful, and may lead to a better understanding of the design process and the issues facing the designer regarding the use of guidelines. If the manner in which guidelines are presented is problematic, then perhaps the answer rests with the development of aids. Should guidelines be presented in a paper-based medium, or should guideline authors use electronic tools to better express guidelines? Or do guidelines, regardless of medium, require a tool through which they might be understood?

2.5.2 Tools for the use of interface design guidelines

The process of developing tools to assist designers with guidelines began in the 1980s

(Campbell, 1996). The need for a tool to improve access to specific guidelines (and to aid in their use) emerged from problems that were identified in the Smith and Mosier (1986) publication. As

Chapter 2 46

discussed in Section 2.2, Guidelines for Designing User Interface Software was one of the first

sets to comprehensively sort through existing research and collect guidelines for computer

system interfaces. The benefits of having such a large number of guidelines in a single publication are tempered by the very expansiveness and sheer size of the report. The

comprehensiveness of including 944 individual guidelines is a potential problem; how can the

designer find the correct guideline and comfortably know that they have found all the necessary

guidance? The use of cross references is of some help to designers using the report, but the

emergence of tools as early as 1988 was meant to take advantage of computer technology to aid

in the manner of presentation of, and access to, guidelines. Iannella (1995) discusses tools

developed following Smith and Mosier’s publication; including Navitext SAM (in 1988) and

BRUIT SAM (in 1992). Each of these tools takes advantage of both hypertext for cross-

references and databases to store guideline information. Navitext SAM offered a framework for

searching the Smith and Mosier guideline set. Interestingly Perlman (1989), the author of

Navitext SAM, noted the importance of using a checklist. Perlman believed that finding, “a

generalizable method for managing checklists is an important concept for a design and

evaluation tool that acknowledges human memory limitations” (Perlman, 1989, p. 67).

The second of these tools, BRUIT SAM, was tested by users who recommended further enhancements which were subsequently incorporated prior to the launch of a version named

HyperSAM (Ionella, 1995). HyperSAM12 was developed to support browsing, gathering, annotating and searching of the Smith and Mosier guidelines. Using a system of framed windows, HyperSAM allows the designer to use the power of a database to find references, to gather disparate as well as related guidelines and to add text annotations to specific guidelines.

12 Iannella states that the “A” in SAM does not refer to “and”, but rather to Arlene Aucella who worked with Smith on an earlier version of the guidelines prior to the involvement of Mosier (Ianella, 1995).

Chapter 2 47

Ionella (1995) points out that HyperSAM is intended to be a shell not dependant on the Smith

and Mosier guidelines. The tool should allow for the updating of guidelines and supporting

material at the database level, while maintaining the integrity of the tool’s interface. Based on the

author’s outline and illustrations, one could anticipate that designers would prefer using such a

tool instead of the over 475 page bound guideline set, but there is no information available

regarding designers’ attitudes.

Beier and Vaughan (2003) used a different approach regarding interface guidelines for

web applications. Instead of developing a tool with which the designer can interact, they

developed a framework through which designers could better access guidelines and understand

the interrelationship of the guideline set. The Bull’s Eye was a tool intended to address the

problem of working with guidelines that were either too general or too specific. The authors grouped guidelines by focus (for example, those that were relatively narrow in focus and addressed a specific component (or widget) in a particular application were placed in one grouping, while guidelines with a broader focus such as those concerning elements common on various pages or interface “screens” were organized into another group. This arrangement was visualized using the metaphor of a bull’s eye; with specific widget-related guidelines in the centre and general principles furthest out (see Appendix B). There were a total of five areas in the construct. While the authors discuss the level of support their approach received from their employer (Oracle), they do not specify how the tool is implemented except to imply that understanding a series of guidelines within a bull’s eye metaphor will result in design consistency and the proper application of guidelines. Unlike HyperSAM, which operated as a tool with which users interact on a screen, the bull’s eye tool is effectively a conceptual

framework within which the guidelines are organized. The company was able to implement this

Chapter 2 48

tool, over a nine month period, to ensure that all their web-based products incorporated existing

web design guidelines (Beier & Vaughan, 2003).

The problem of applying general guidelines to context-specific problems led to the development of a tool which Henninger, Haynes, and Reith (1995) felt would take HyperSAM a

step further, coupling the guidelines with a history of context specific cases, representing an

“organizational memory”. In two papers published over a seven year period, the authors

proposed linking specific guidelines with cases; specific examples that could grow with the use of the tool itself (Henninger, Lu, & Faith, 1997; Henninger, Haynes, & Reith, 1995). As the number of cases increases, so does the usefulness of the tool, providing designers with more

context specific examples of how to best apply a guideline. The authors suggest the deployment

of a process within organizations to ensure a post design implementation survey be developed

and incorporated into the tool and to formalize the process of project tracking and reporting

which can then be linked to the specific guidelines. The long term goal of the project was

creating a repository of cases which would highlight which guidelines work best in a given

context. The concept of using organizational knowledge and memory and identifying and

ranking guidelines based on a record of successful design efforts represents not only the potential

for an excellent tool in the aid of using guidelines, but a tool for sharing knowledge regarding

good design. Unfortunately the authors of this paper do not return to this idea in future

publications. The work suggests an idea that could be further developed, creating yet another

opportunity for future researchers in the area of web interface design guidelines.

Specifically considering eBusiness web sites (which the paper defines more broadly than

e-commerce, as the use of electronic communication to conduct business), Head and Hassanein

(2002) collect a number of applicable guidelines. The guidelines are all drawn from existing

Chapter 2 49 sources and divided into three categories; site level, page level, and content level. While the manual evaluation of eBusiness sites is discussed, the authors consider nine tools that were commercially available (at the time of publication) to assess a site’s adherence to guidelines. The paper, written as a general overview, recommends that a combination of automated and manual evaluation is ideal (as opposed to using just one approach.) But perhaps the greatest value of this publication with regards to guideline related research is in the recommendation the authors make for future research. Head and Hassanein (2002) suggest more research in the area of how different guidelines impact usability with the goal of developing a quantifiable weighted model.

This suggestion highlights the need to better understand the effect of differing guidelines.

By 2005, the prevalence of both multimedia in web-based interface design and design tools led Sutcliffe, Kurniawan and Shin (2005) to develop both a design method specific to multimedia and an accompanying tool to assist with implementation. The multimedia method uses a variation of a cognitive walkthrough to better understand the information needs of users. From this data, an information tree is developed, which further spawns a task model. The authors first present a short series of guidelines (which they state is synthesized from various sources) along with other details from the proposed guideline application method, and then discuss how the design method was tested. The study involved nineteen students and showed improved design results from those subjects who made an attempt to use the design method in order to first recognize, and then apply, the correct guidelines. The authors do caution that while the, “qualitative evidence indicates that the guidelines do have a positive impact on solution quality we cannot rule out that good quality solutions might be produced by talented individuals without such advice” (p. 390.) Clearly, further study is necessary to explore the effects of other variables on the design solution.

Chapter 2 50

Kim (2010) looked to better understand the design strategies of designers in order to

more effectively organize design guidelines. The research first considered designers’ behaviours

and discussed the relationship between the design process and access to relevant guidelines;

implying the importance of accessing the right guideline at the right time. Kim argued that while

guidelines are helpful for designers, the presentation of the guidelines can be improved by

ensuring multidimensional organization – an organization which allows designers to find the

relevant guideline either through the perspective of the designer (through recognition of a design

pattern or a search for the appropriate rule) or from the perspective of potential users. The former

emerges from the experience of the designer, whereas the latter is based on the heuristic

evaluation that design might encompass. In testing these assumptions, Kim’s approach (a three-

step method to apply to a design problem) improved the time it took participant designers to find

the appropriate guideline.

Research which either explores the presentation of guidelines or the development of tools

to assist in the use of guidelines both address means through which guidelines can be better used by designers and evaluators. The next section looks at research which explores whether compliance with interface design guidelines is related to improved user performance.

2.5.3 The relationship between user performance and compliance with interface

design guidelines

In 1998, a study exploring the conformity of bibliographic displays to guidelines suggested a subsequent study correlating user performance and compliance with display guidelines (Cherry,

1998). Cherry returns to this question in 2006, but not before the publication of two other studies exploring this relationship (D’Angelo & Twining, 2000; Ray, 2002).

Chapter 2 51

The D’Angelo Standards is a checklist of design standards developed by D’Angelo and

Twining (2000). These standards were developed by comparing characteristics of award winning web sites in four categories (academic, business, personal and professional) and incorporated into subsequent research, which set out to test the correlation between compliance to the standards and user performance (measured in time), comprehension and preference. Fifty participants viewed 20 individual screenshots on a computer monitor. After an unlimited period of time, users clicked to change the page and were then asked to answer four multiple-choice questions. The user data was then correlated with the number of D’Angelo Standards contained in each display. The authors report a weak, but positive (0.072) correlation between time and number of standards and a weak but also positive (0.266) correlation between comprehension and number of standards. They make no specific mention of statistical significance regarding the study findings. With regards to preferences, the displays least favoured by participants complied with fewer of the D’Angelo Standards than did the most favoured displays.

While the study is notable as the first to investigate the relationship between conformity to a guideline set and user performance, some minor issues with the methodology should be noted. Having a set period of time to view each interface may have strengthened the comprehension measure. The authors mention considering the relationship between the variables of time, comprehension and the number of standards, but do not provide details beyond reporting a positive correlation. However, the preference measure is useful, since emotional reactions to a site have been shown to affect surfing behaviour (De Angeli, Sutcliffe, & Hartmann, 2006; Kim,

Lee, & Choi, 2003; Norman, 2004).

Ray’s (2002) doctoral dissertation explored differences in user performance over time with four web displays. A guideline set was derived from Nielsen’s Designing for Web Usability

Chapter 2 52

(2000). Specifically, the author adopted guidelines pertaining to (i) frames, (ii) navigation

visibility, (iii) feedback, (iv) path length13 and (v) colour. Using this guideline set, Ray designed

four web displays; two of which conform to the guidelines (and which he labels Ideal) and

another two of which contradict the guidelines (which he labels Non Ideal). The research

examined the following hypotheses; (i) that user performance would be superior when

interacting with the Ideal site as opposed to the Non Ideal site, (ii) that repeated use of the Non

Ideal site interface (on which it is assumed participants will have performed poorly) would lead

to improvement over time, and (iii) user performance will improve when participants use the

Ideal site in the final session, indicating that conformance to design guidelines enhanced

usability. Ray expected users to show a preference for the Ideal over the Non Ideal sites. With a

few exceptions attributed to task difficulty, the research supported the contention that

performance would improve using the Ideal site over the Non Ideal site with respect to the

measured variables. As well, the qualitative assessment of user satisfaction found a preference

for the Ideal sites.

A pilot study conducted by Ray suggested that variations in user performance are not the

result of a single guideline, but rather a set of guidelines and the main issue facing site usability

is which combination of guidelines is included in this set. Ray suggests that the problems with

the user performance measures on the Ideal site were a result of poor guidelines which needed to

be revised before improved performance could be observed. The fine tuning of research tools is

understandable, but the need to revise the guidelines leads to asking what was wrong with the

original guidelines and how would a designer, in the course of work, know that a guideline needs

revision? This further leads to a need to better understand when a designer would consult a guideline or more specifically, to a better understanding of the user of the guidelines; the

13 “path length” is a measure of the number of links the user must traverse to find a specific piece of information.

Chapter 2 53

designer. Before shifting focus, we should consider a study with perhaps the most unsettling

results regarding the question of whether conformity to design guidelines correlates with user performance.

Published in 2006, Cherry, Muter, and Szigeti used performance data and data from an evaluation of bibliographic displays to explore the relationship between guideline conformity and user performance. The study involved user performance data collected from 27 participants interacting with nine bibliographic displays. The nine bibliographic displays were scored using two sets of guidelines, the first developed from the literature, the second using 73 guidelines rated as having either moderate or strong research support and drawn from two of the Research- based Web Design and Usability Guidelines sets (Koyani, Bailey, & Nall, 2003; National Cancer

Institute, 2001), as well as ISO 92411-11, 12 and 14 (ISO, 1998, 1997a, 1997b).

The main study correlated user performance with the scoring of the bibliographic displays. Seventeen percent of correlations between response times and guideline conformance scores were significant at the .05 level and indicated slower response times with higher conformity to guidelines. A supplementary study also reported weak correlations between the various response times and the interface’s level of conformity to the guideline set. Only one of the 24 correlations was significant at the .05 level, and indicated slower response times with higher guideline conformity. The authors conclude by stating that a finding of zero correlation between user performance and conformity to guidelines is not as puzzling as is the discovery of a negative correlation. The unexpected findings might be a result of higher level interactions among the guidelines, and suggest that the use of guidelines may be problematic, although the study is certainly not conclusive in this regard.

Chapter 2 54

Considering these four studies, one cannot make conclusive statements regarding the

relationship between user performance and conformity to design guidelines. Instead, the studies

identify an area of research which requires further attention. The authors of the Research-based

Web Design & Usability Guidelines suggest that their guideline set is aimed at, “Web site managers, designers, and other involved in the creation or maintenance of Web sites” (U.S.

Department of Health and Human Services, 2006, p. xv). The next section considers research which explores the background and information needs of designers.

2.6 Research regarding the users of guidelines: web interface designers

Clarifying the intended audience for guideline sets is a complex task. As a descriptive label, web interface designers casts a wide net. Individuals involved in the design of web sites defy narrow description in part because the ubiquity and variety of website development requires a wide set of skills. Web sites can be relatively simple or very complex, and the skills required for either simplicity or complexity varies (Connell, 2008).

Citing the importance of academic library web sites to the services a library offers,

Connell surveyed 110 web developers regarding their background and use of tools in their work.

As expected by the author, experience and educational levels varied among respondents, as did the average length of the design process at their particular institution. The survey found that the ability to “organize information effectively was the most important quality” a developer working in an academic library could possess (p. 129).

Karam, O’Connell and Perry (2009) found knowledge sharing between design engineers

occurred predominantly via face to face contact. Online communities and forums were perceived

as too time consuming. Carvalho and Dong (2009) compared different design disciplines and

Chapter 2 55 found that not only is different knowledge required for each discipline (which they expected to find), but that the differences in how knowledge was valued was a key factor in both understanding how communities viewed their members and in how individuals in different disciplines viewed each other. “Design” was found to be a malleable term. Surveys of designers

(Salter & Gann, 2003; Hunkleler & Vanakair, 2000) and interviews with design practitioners

(Michlewski, 2008; Carvalho & Dong, 2009) reinforce the difficulty in defining characteristics of this disparate group. But this does not make the endeavour without merit. The design process is complex, and the more knowledge gained about designers, the better our understanding of how to best develop tools or aids to help in the design process.

Tang and Aleti (2010) argue that design is a “rational problem solving activity – it is a process in which designers explore the design issues and search for rational solutions” (p. 630)

Tang and Aleti’s study focused on software design, but their conclusions that effective design emerges from good planning, well organized problem exploration and good reasoning speaks to the complexity of the design process. The authors do not discuss the use of tools such as guidelines, but they do suggest methods to aid in the making of design decisions.

Challenges faced by designers can be framed as design trade-offs; the choices made by designers in the process of design. More specifically, they are the choices made between two valid design solutions. Studies have explored four trade-offs:

1. Ensuring site accessibility for most potential uses and the use of graphics,

images and multimedia elements within a page. (Regan, 2004)

Chapter 2 56

2. Ensuring site accessibility while incorporating screen based controls or widgets,

which operate outside of browser based controls. (Regan, 2004; W3C, 2008)

3. Optimizing the user experience while addressing the needs of content using

either scrolling or paging as a means of display. (Norman, 1983)

4. The layout of a page so that the user can find the content they seek versus

broader content organization concerns. (Veen, 2001)

These trade-offs all speak to the contextual challenges of design where the needs of various users must be addressed in the design of a page.

2.7 Discussion and related research questions

The literature in web-based interface design guidelines has moved from the need to compile a comprehensive guideline collection (Smith & Mosier, 1986) to questioning whether compliance with guidelines can lead to improved user performance (Cherry, Muter, & Szigeti, 2006). How guidelines themselves are defined has also broadened from the first publication of Smith and

Mosier’s set (1986) to the growth of pattern libraries which operate as guideline-like aids to designers (Seffah, 2010). The research reported in the literature can be grouped into three broad categories; the translation of published research into guidelines, the use of guidelines in the

design and evaluation process, and the interaction of guidelines with other guidelines.

The interaction between guidelines is an area that has generated the least research

interest, but which may hold the key to understanding how guidelines may best be structured in

terms of format and used on an operational level. Yet the challenge of understanding this

Chapter 2 57

interaction is enormous. Smith and Mosier (1986) questioned whether a comprehensive set of

guidelines could ever be derived from experimental data. The concern that testing even a small

number of interacting variables would require experimental design of great complexity is no less valid today.

Whether guidelines are the best tools for codifying knowledge remains unknown.

Literature shows that the relationship between user performance and guideline compliance is problematic (Cherry, Muter, & Szigeti, 2006; Cherry, 1998), but whether designers are aware of such problems is unclear. Efforts to codify knowledge in the form of guidelines would benefit from better understanding how designers view guidelines and addressing the question; what are web interface designers’ attitudes towards the use of design guidelines?

The relationship between research and guidelines is explicit when considering evidence- based guidelines, where the authors of the compilations have translated and synthesized research in order to articulate guidance for those involved in the design process (U.S. Department of

Health and Human Services, 2006; ISO 1998, 1997a, 1997b; Koyani, Bailey, & Nall, 2003;

National Cancer Institute, 2001; Shneiderman & Plaisant, 2009; Shneiderman, 1987; Smith &

Mosier, 1986). This is an important relationship, since cited research has the potential of reassuring guideline users that recognized research practices stand behind the advice. Such a guideline is not the opinion of a single expert, but is based on rigorous scientific evaluation. Yet the degree to which such guideline sets are used by the design community is unknown, as is the community’s awareness of such sets. A first step in understanding the role of evidence-based guidelines in design is to address the question: what is the level of awareness and usage of evidence-based web design guidelines by web interface designers?

Chapter 2 58

The use of research in guideline development has not been adequately explored in the literature. There is value in ensuring that guidelines which cite published research are in fact accurate translations of the research. Research which is relevant to web interface design can be complex and reducing findings to a few sentences of advice risks losing the nuances and subtleties of good research. In addition, the interpretation of such guidelines needs to be better understood, since the reduction of research to a guideline may lead to variations in interpretation.

Specifically, how are evidence‐based web design guidelines interpreted by web interface designers?

The literature has numerous examples of research considering problems with how the guidelines are understood and applied by those involved in the design process (Ivory & Megraw,

2005; Ivory, Sinha, & Hearst, 2001; Vanderdonckt et al. 2000; Tetzlaff & Schwartz, 1991), tools that would better facilitate this process (Sutcliffe, Kurniawan, & Shin 2005; Beier & Vaughan,

2003; Henninger, Lu, & Faith, 1997; Henninger, Haynes, & Reith, 1995; Ionella, 1995, Perlman,

1989; Cohen et al., 1994), research that considers the effect of guidelines usage on user performance (Cherry, Muter, & Szigeti, 2006; Ray, 2002; D’Angelo & Twining, 2000; Cherry,

1998), and studies that look to better understand the designer and the design process (Carvalho &

Dong, 2009; Connell, 2008; Hunkleler & Vanakair, 2000; Michlewski, 2008; Salter & Gann,

2003; Tang & Aleti, 2010). While the relative volume of research considering how best to use guidelines in the design process is high, the literature does not lead to conclusive findings.

Further, it is not known where in the design process guidelines are being consulted. Clarifying when guidelines are used may help us to understand the role guidelines play in design. Defining whether guidelines are, for example, prescriptive, supportive or corroborative, might aid authors in ensuring that guidelines better meet the needs of the design community. Further research is

Chapter 2 59

required to answer the question; when are guidelines used in the design process to solve design

problems?

The review of the literature highlights a number of questions which, if answered, will lead to a better understanding of how guidelines might better address web interface design problems. Identifying the level of awareness and the opinions designers hold regarding design guidelines will help us understand the role that guidelines currently play in web interface design.

Those involved in all aspects of web interface design would benefit from a clear understanding of guideline use, from identifying where in the design process guidelines are used and from clarifying if guidelines are interpreted consistently by designers. In addition, we currently do not know what specific design trade-offs influence the work of designers or how design knowledge is shared between web interface designers. Understanding how knowledge is shared and the role web interface design guidelines play in resolving design problems (including trade-offs) will help us identify not only areas of future research, but tools which can aid in the design process.

To reiterate, six research sub-questions, which remain unanswered in the existing literature, are explored in this dissertation:

RQ1. What are web interface designers’ attitudes towards the use of design guidelines?

RQ2. What is the level of awareness and usage of evidence-based web design guidelines by

web interface designers?

RQ3. How are evidence-based web design guidelines interpreted by web interface designers?

RQ4. When are guidelines used in the design process to solve design problems?

RQ5. What specific design trade-offs influence the work of designers?

RQ6. How is knowledge and information shared amongst web interface designers?

Chapter 2 60

Chapter 3

The application of interface design guidelines (Study #1)

3.1 Introduction

In order to better understand the web interface design process, researchers have considered the

relationship between the design process, access to relevant guidelines and the behaviour of

designers (Kim, 2010; Ivory & Megraw, 2005; Tetzlaff & Schwartz, 1991). The literature

suggests that guideline sets are welcomed by designers, but how best to incorporate the design advice provided in guidelines is unclear. Two of the nine participant designers in Tetzlaff and

Schwartz’s study (1991), described the use of guidelines in the course of the study to be an artificial process; a method forced upon them by the researcher, as opposed to a method they would normally use. The designers in the study reported a preference not to use guidelines in their actual work (Tetzlaff & Schwartz, 1991). It remains unclear why designers did not consult guidelines in the course of their work, or if the results reported in the literature represent a broader sample of web interface designers.

The first study, described in this chapter, explores the attitude of web interface designers regarding the application of guidelines to design work. Specifically, the study considers three research questions:

RQ1. What are web interface designers’ attitudes towards the use of design guidelines?

RQ3. How are evidence-based web design guidelines interpreted by web interface

designers?

RQ4. When are guidelines used in the design process to solve design problems?

Chapter 3 61

3.2 Methodology

The study asked a group of engineering students to use a guideline set in the design of a Wiki page. This task involved the participants familiarizing themselves with the guidelines, potentially using the guidelines in the course of their design work, and recording whether or not they found each of the guidelines applicable to their work. They were also asked to provide either a description of how the guideline was applied, or a rationale for why they felt the guideline was not applicable.

3.2.1 Method

Using 25 guidelines drawn from the Research-based Web Design & Usability Guidelines (U.S.

Department of Health and Human Services, 2006), 16 participants were asked to consider the guidelines in the context of a design project conducted during a 13-week course. The guidelines were distributed to the participants as they began to initially consider their design projects.

Working in groups of three or four individuals, participants were tasked – within the context of the interface design course – to develop a web-site within a Wiki environment. Their use of the guideline set was voluntary and was not considered a factor in the grading of their project.

Participants received a thirteen-page booklet containing 25 guidelines and a document which listed the guidelines and provided space for the description of usage or rationale for not using a particular guideline. After completing and returning this document, the participants participated in an hour-long focus group session. Participants were divided into two groups to better facilitate focus group discussion regarding their experiences.

Chapter 3 62

3.2.2. Apparatus

The study used a selection of guidelines from Research-based Web Design & Usability

Guidelines (US Department of Health and Human Services, 2006). The original publication,

which contains 209 guidelines, was abbreviated by two factors. Only guideline categories which

were (i) deemed appropriate for the specific task of designing with a Wiki environment, and (ii)

which were supported by either moderate or strong research support, were included. The Wiki environment imposes numerous restrictions on designers. While Wiki pages can be designed by those with limited or no prior web design experience, there are design elements that cannot be applied within the Wiki environment. Not only was there an attempt to remove guidelines that would be irrelevant in a Wiki environment, but reducing the total number guidelines also ensured that designers would not be overburdened with too many guidelines to consider, which has been identified as a problem in previous research (Kim, 2010; Smith & Mosier, 1986; Vanderdonckt et al., 2000). The three categories in the booklet were Page Layout, Text Appearance, and

Graphics, Images and Multimedia. The first category provides guidance to ensure comprehension of the content on a page and to provide the user of the web site with design consistency. The second category, with seven guidelines, is focused on the appearance of text on a page. The final category, comprised of ten guidelines, speaks to the use of images or multimedia (non-textual content) in a web page. A brief introduction to the categories, as well as the 25 guidelines, were collected in booklet form and distributed to participants.

Four weeks after receiving the guideline booklet, participants received a document in which to record their application (if any) of a guideline to their Wiki page design. The document listed each of the 25 guidelines, including a brief guideline description and two sections where feedback was requested. The first section asked participants to indicate, via a checkbox, whether

Chapter 3 63

the guideline had been APPLIED TO DESIGN, PARTIALLY APPLIED TO DESIGN or NOT APPLIED TO

DESIGN. The second section asked participants to describe their rationale either via description of

its use, an explanation of why only part of the guideline was used and how, or an explanation of

why a guideline was not used (see Appendix C).

3.2.3 Participants

The participant designers were students enrolled in an engineering course at a Canadian

university. Of the 16 participants, six were women and ten were men. Seven of the participants

reported previous experience with interface design. The goals of the group project were to design

an ergonomic Wiki page that meets the requirements of various users and is sufficiently easy to

use. As part of the project, the group members were identified in one of three roles; designer,

information specialist or researcher. These titles represented different responsibilities, although all members were responsible for the design of the final Wiki page. This meant that each member had to consider design issues and whether use of a guideline was appropriate to their task, regardless of their specific role.

Regarding consent, the researcher had no prior relationship with the students and the course instructor did not play any role in the study, with the exception of allowing the study

introduction, collection of booklets and documents and the focus group sessions to occur during

course time. Consent was given by 16 of the 18 students in the class. The two students who did

not provide consent did not play any role in the study.1

1 The course instructor did not know who participated in the study. The two students who did not provide consent were welcome to use the guidelines in their course work, but they did not contribute data or participate in the focus group discussions. The study protocol was approved by the Ethics Review Office of the University of Toronto. Students who did not participate in the study continued to work at their computer stations during the focus groups sessions. Note that participants were not compensated.

Chapter 3 64

3.2.4 Procedure

Following an explanation of the study and distribution of consent forms, the booklet was

distributed to the participants during their regular class period. Four weeks later participants

received a document in which to record their application (if any) of guidelines to their project.

When the class met a week later, this document was collected from the students and preliminary

analysis was conducted in preparation for focus group sessions held the following week. Due to

the number of participants, two separate focus group sessions were held (see Appendix D for

focus group discussion script). A table at the front of the large classroom was used to conduct the

first focus group session, while the remaining students worked at computer stations throughout

the class. The size of the laboratory classroom (which included outside students working at

computer terminals), allowed for a measure of privacy at the front of the room, where the focus

group sessions were conducted. During the second hour, the positions were reversed. This

allowed the focus group sessions to be conducted in sequence. The sessions were audio recorded.

In addition to returning the document containing assessments, participants were invited to return their guideline booklets if they contained any marginalia (in exchange for a clean copy).

3.2.5 Measures

The study captured the number of guidelines applied, partially applied or not applied. As well,

participants included a rationale for the decision to use or reject a guideline, thus providing more

information regarding guideline use. In addition, the recordings of the focus group sessions

provided insight into the rationale for their application (or non application) of guidelines.

Chapter 3 65

3.3 Results

Data was collected in two formats; documents returned by participants indicating guideline use

(or non-use) during the design process and transcripts of recorded focus group sessions. The two types of data will be discussed below; first the data collected in the documents, and then the transcription of the focus group sessions.

3.3.1 Guideline use in the design of projects

Documents completed and returned by the 16 participants reported the use (or non-use) of specific guidelines with accompanying rationale. Only two of the guidelines found universal application, whereas other guidelines were rarely applied. Variations in the use of guidelines were expected, given the breadth of guidance found across the 25 guidelines.

Sixteen participants considering 25 guidelines led to a total of 400 instances (16 participants x 25 guidelines) where a participant might apply, partially apply, or not apply, a guideline. In 51.8% of the instances a guideline was applied to the Wiki page design. In 22.5% of the instances the guideline was partially applied and in 25.8% of the instances the guideline was not applied to the design. Table 3.1 lists the guideline ID#, the guideline, and the number of instances that the guideline was applied, partially applied or not applied to the Wiki page design.

Chapter 3 66

Table 3.1: Application of guidelines by 16 participants in design project

ID# Guideline Applied Partially Not applied applied Page Layout (from Chapter 6 of U.S. Department of Health and Human Services, 2006 guideline set) 6:2 Put important, clickable items in the same locations, and closer to the top of the page, 9 7 0 where their location can be better estimated. 6:3 Put the most important items at the top center of the Web page to facilitate users’ finding 7 6 3 the information. 6:4 Structure pages so that items can be easily compared when users must analyze those 3 5 8 items to discern similarities, differences, trends, and relationships. 6:7 Visually align page elements, either vertically or horizontally. 16 0 0 6:9 Ensure that the location of headings and other page elements does not create the illusion 8 7 1 that users have reached the top or bottom of a page when they have not. 6:11 Limit the amount of white space (areas without text, graphics, etc.) on pages that are used 10 5 1 for scanning and searching. 6:12 If reading speed is most important, use longer line lengths (75‐100 characters per line). If 8 7 1 acceptance of the Web site is most important, use shorter line lengths (fifty characters per line). 6:13 Use frames when certain functions must remain visible on the screen as the user accesses 8 2 6 other information on the site. Text Appearance (from Chapter 11 of U.S. Department of Health and Human Services, 2006 guideline set) 11:1 When users are expected to rapidly read and understand prose text, use black text on a 15 0 1 plain, high‐contrast, non‐patterned background. 11:4 Ensure visual consistency of Web site elements within and between Web pages. 13 3 0 11:6 Use attention‐attracting features with caution and only when they are highly relevant. 8 6 2 11:7 Use a familiar font to achieve the best possible reading speed. 16 0 0 11:8 Use at least a 12‐point font (e.g., typeface) on all Web pages. 13 3 0 11:9 When using color‐coding on your Web site, be sure that the coding scheme can be quickly 7 3 6 and easily understood. 11:10 Change the font characteristics to emphasize the importance of a word or short phrase. 11 3 2 Graphics, Images and Multimedia (from Chapter 14 of U.S. Department of Health and Human Services, 2006 guideline set) 14:1 Use background images sparingly and make sure they are simple, especially if they are 7 0 9 used behind text. 14:2 Ensure that all clickable images are either labeled or readily understood by typical users. 11 2 3 14:3 Take steps to ensure that images on the Web site do not slow page download times 10 4 2 unnecessarily. 14:4 Use video, animation, and audio only when they help to convey, or are supportive of, the 4 4 8 Web site’s message or other content. 14:5 Place your organization’s logo in a consistent place on every page. 2 1 13 14:6 Do not make important images look like banner advertisements or gratuitous decorations. 6 3 7 14:10 Include actual data values with graphical displays of data when precise reading of the data 3 2 11 is required. 14:11 Use a graphic format to display data when users must monitor changing data. 1 3 12 14:13 Use images that look like real‐world items when appropriate. 4 6 6 14:15 To facilitate learning, use images rather than text whenever possible. 7 8 1 Total instances 207 90 103

Chapter 3 67

Of the 25 guidelines, only two were applied by all participants: Guideline 6:7 and Guideline

11:7. Guideline 6:7 addresses the visual alignment of content on a page, whereas Guideline 11:7 suggests using a familiar font to ensure optimal reading speed. In terms of guidelines less used, only one participant applied guideline 14:11 to their Wiki page, a guideline which speaks to the need to use a graphic format (instead of text) when users are required to monitor changing data.

Of the 25 guidelines, 14 were applied by a majority of participants. Table 3.1 illustrates differences in the application of guidelines; five guidelines (the two mentioned above, 6:7, 11:7, as well as guideline 11:1, 11:4 and 11:8) were applied by at least 13 participants. Applications of the other 20 guidelines vary to differing degrees.

In terms of category differentiation, Table 3.2 illustrates that guidelines within the Text

Appearance category were applied by participants in 74% of possible instances. Guidelines within the Page Layout category were applied in 54% of instances and guidelines within the

Graphics, Images and Multimedia category were applied least often (35% of instances).

Table 3.2: Guideline usage by category

Categories % of instances guidelines in a category were used

Applied Partially applied Not applied Page Layout 54% 30% 16% Text Appearance 74% 16% 10% Graphics, Images and Multimedia 35% 21% 44%

Chapter 3 68

The Wiki page design limited the use of images and multimedia, which may explain the lower number of applied guidelines. Discussion in the focus group sessions supported both the idea that the Wiki page design imposed limitations on the application of certain guidelines and that the interpretation of guidelines was problematic.

The reasons reported in the assessment document for not applying a guideline fall into one of two broad categories, (i) perceived technical restrictions of the Wiki environment or (ii) the guideline spoke to a design feature not present in the participant’s design. In some cases, the participants did not provide sufficient detail, such as when the participant wrote, “does not apply” as the reason for checking off the box marked does not apply. Table 3.3 reports the reasons given that guidelines were not applied to the design, including a category (Other) where insufficient or no information was provided by the participant as to the reason.

Table 3.3: Reasons guidelines were not applied to the design

Reason given why guideline was not applied % of respondents

Focus of guideline was not applicable 73% (context dependent) Technical restrictions of the Wiki environment 21% Other 6%

Almost three quarters of the instances where a guideline was not applied was due to the irrelevance of a guideline to the design. A number of guidelines were context dependent, which precluded their use. In particular, Guidelines 14:10 and 14:11 spoke to the use of graphics and information visualizations on the page, yet a number of participants proposed designs which did not use graphics or information . This was expected due to the limitations of the

Chapter 3 69

Wiki environment. The contextual relevance of a guideline was open to interpretation. For

example, Guideline 6:13 described aspects of frame use in a page. The guideline was only

applicable if the designer elected to use frames in their page design. Where the page designs did

not incorporate frames, this guideline was obviously not applied. In another example, Guideline

14:1 calls for the designer to “Use background images sparingly and make sure they are simple,

especially if they are used behind text”, but if the designer has elected not to use a background

image, are they in fact complying with a guideline that suggests the use of such images

“sparingly” or did the decision predate the consideration of the guideline (i.e., the designer

elected not to use a background image, and therefore deemed that the guideline was not applied)?

For one participant, the decision was based on their perceived convention of a Wiki page. They

wrote, “To maintain the standard convention of Wiki entries, we did not utilize a background image. A standard white background was utilized instead.”

Perceived limitations of the Wiki environment accounted for 21% of the reasons.

Regarding Guideline 14:5 (Place your organization’s logo in a consistent placed on every page), one participant wrote that the guideline could not be applied because a Wiki logo already rests in a consistent location on each page and that “Wikipedia is the organization’s logo, so I have nothing to do with it.” Another participant questioned the validity of a guideline and did not apply the guideline because they felt it would hinder the design. In response to Guideline 11:10

(“Change the font characteristics to emphasize the importance of a word or short phrase”) the participant felt following this guideline would be detrimental and would “reduce reading speed.”

Further reasons as to why the guideline would reduce reading speed were not provided. The guideline was not applied.

Chapter 3 70

In 6% of instances the reason was either left blank or the participant wrote “Does not

apply.” The documents also revealed at least one misunderstanding (where a participant checked

NOT APPLIED, but in the explanation indicated that they had used the guideline.) The low reporting of problems in the document regarding interpretation of guidelines ran counter to the data collected in the focus group sessions, where a larger number of guidelines were identified as being confusing, unclear or difficult to interpret. The reasons for the discrepancy are unclear and perhaps have to do with the tools used in the study or the nature of a focus group session, where peers influence discussion (Sim, 1998; Lunt & Livingstone, 1996). The discussion may have highlighted problems or issues that were not perceived or articulated by participants while considering the use of guidelines in isolation.

3.3.2 Focus group sessions

Two focus group sessions were held; the first with nine participants (comprised of three design project groups of three students each), and the second with seven participants (comprised of two groups, one with three members, the other with four) for a total of sixteen participants. The design project groups were kept together in the hope that the dynamic between members would shed light on the interpretation of guidelines, a factor discussed in previous studies (Ivory &

Megraw, 2005; Cherry, Muter & Szigeti, 2006). Each session started with brief introductions and

initial thoughts on the guideline set, and in both sessions discussion immediately turned to the

technical limitations of the Wiki environment and the difficulties in applying a number of the

guidelines. Similarities between the two focus group sessions were found in discussing a number

of key areas; interpretation of guidelines, role of cited research, and where guidelines could be best used in the design process.

Chapter 3 71

While the participants were not familiar with the source of the 25 guidelines (Research- based Web Design & Usability Guidelines, U.S. Department of Health and Human Services,

2006) they did express familiarity with the guideline contents. The term used to describe them was “common sense.” One participant stated that they, “…found that many of the guidelines already applied to what I was originally going to do. It seemed like common sense.” This description of some guidelines as common sense was formed by participants’ previous experience as designers, users of web-based interfaces, or both. As one participant indicated,

“we’ve had exposure to a lot of these guidelines before so for us it’s second nature.” The exposure reported by participants was not to the specific set used in the study, but rather to similar ideas found in other literature.

Familiarity with similar ideas was a result of both visiting web-sites or from other classes, where, “these ideas have been hammered into our heads for the past two years so for a lot of it we don’t need the guidelines to know what to talk about. If you hadn’t done design it wouldn’t be so obvious.” A number of participants had studied human factors issues within a design context and brought that knowledge to bear on their design of a Wiki page. They made reference to specific classes and instructors who had first presented some of the ideas captured in the guideline set used in the study. Seven of the participants described experience specifically as interface designers (which involved designing websites using HTML coding), but all reported comfort with assessing design usability due to previous academic work. Participants considered themselves “experts”, in contrast to the average user. Such perceived differences in usability experience were mentioned throughout the focus group discussions. Ideal users of the guideline set, in the words of one participant, were “people who design websites [but] aren’t exposed to a fraction of the human factors design experience that we have been.” They might not be termed

Chapter 3 72

“novice” users, but rather people, “without user interface design training and the [related] history and background.” Other participants used different expressions to describe the ideal user of guidelines. The common definition, regardless of the descriptive terms used, was that the ideal user lacked usability or human factors . The participants in the focus group sessions felt they had this knowledge and argued that the guideline set was of less value to them than to those without the same education or background. “I definitely think it’s useful for training purposes and for anybody lacking experience designing,” said one participant. This belief was echoed by other participants.

Yet familiarity with the ideas captured in the guidelines did not mean clarity. The recommendations captured in Guideline 14:11, which instructed designers to use a graphic format in the display of data, were considered too brief to be understood by some participants.

Even the use of visual examples in the guideline set did not always clarify the approach that the designers should take to comply with the guideline. Based on participants’ reactions, the most problematic guideline was 6:11 which provided guidance on the use of white space in a page layout. Ten participants indicated that they applied this guideline, five partially applied the guideline and one did not. This guideline came under considerable criticism during the first focus group session for its perceived subjectivity, in that “limit the amount of white space on pages that are used for scanning and searching” was thought to be too vague. As one participant noted,

“I think to everyone that could be different. You can’t quantify that number, so it would be useful if you said something like if out of this percentage of the page is content, this percentage should be white space or something like that.” Participants in the second focus group session reported even stronger disagreement with the guideline. A number of participants did not understand the guideline itself. “What adverse effect does the white space have on scanning or

Chapter 3 73

searching?” they asked. During the discussion, participants were clearly unconvinced that

Guideline 6:11 was worth considering in the context of their design work:

[Participant A] … if you’re searching for stuff, wouldn’t you want less stuff on

the page and more white space so you can discern what you are looking for

easier?

[Participant B] … Yeah, they say limit, which implies that it’s bad for searching.

But it’s the opposite. You’d want more white space.

[Participant A] … It’s implying that white space is bad for searching, but

intuitively for me it’s the opposite.

[Participant C] … The guideline seemed bizarre.

[Participant D] … They say that the more you have on the page the less scrolling

that you have to do. But there would be a lot of visual clutter you have to search

through. It would require more cognitive resources to find what you’re looking

for than just scrolling.

However, in the assessment document, 63% of the participants reported applying Guideline 6:11 to their design and only one participant reported they did not apply the guideline. This clear disconnect between their concern with the guideline and their reported use of the guideline is

Chapter 3 74

curious. The ideas informing the guideline resonated with the participants, while the specific guideline left them wanting more detail and advice. This suggests a possible difference between

considering the guideline in isolation (without a specific design problem to address), and trying

to apply the guideline to solve a specific design problem. But such differences, if they exist, are

problematic. An evidence-based guideline, particularly one supported by moderate or strong

research, should provide the user with clarity. The user’s focus should ideally be on considering

whether or not to apply the guideline to a particular design problem, not on trying to interpret the

guideline itself.

The discussion regarding the interpretation of Guideline 6:11 underlined a problem; there

are many ways to interpret a particular guideline. Participants suggested that what was intended

by the authors of the guideline set was not always clear and that “the problem is similar with

other guidelines as well. Basically they’re not specific enough.” Participants in both focus group

sessions found the lack of specificity of guidelines to be problematic, where more detail in the

guideline would be helpful, but only to a point. The participants did not believe that design rules

were needed – and that increasing the specificity of a guideline risked removing the flexibility

that design required. One participant argued for the importance of the creative process in design,

for it was here that “…creativity comes in, right? Just following the same thing would be

limiting.” This statement was echoed by another participant who said that, “…it gets to a point

where it’s either me and my creativity and my design, or sticking to a simple guideline. I would

take my design with some influence from the guideline.”

Participants also had issues interpreting other guidelines. One participant struggled with

understanding Guideline 6:7 (“Visually align page elements, either vertically or horizontally”),

stating that “It took me awhile to sort it out” in part because the examples illustrating the

Chapter 3 75

guideline were unclear. The purpose of the examples within guideline descriptions is ideally to clarify, yet that is not always the case. As one participant noted regarding another guideline,

Guideline 11:7 says use a familiar font. With 1000 fonts out there, I don’t know

what a familiar font is, I would automatically assume it’s Arial or Times New

Roman, but for someone who has never read a guideline before or who would

know what a good web page is, it doesn’t really explain that.

When asked if the examples listed within the guideline clarified the notion of a familiar font, participants responded negatively, and one stated that “… a lot of fonts have been created that could be considered familiar but aren’t familiar.” For the participants, it was difficult to interpret the guideline. The frustration regarding how to interpret the guidelines was evident in other

instances, as when one participant noted that Guideline 6:4, which uses the phrase “can be easily

compared” was, “…really ridiculous. It can be anything, because easy to you might not be easy

to me or to a 60 year old, right?”

Every guideline in the booklet contained reference to supporting research. One possible

solution to a lack of specificity – or lack of clarity – is to consult the cited research. But none of

the participants did this. They stated that not only did they did not have the time to look further,

but that the research evidence was drawn from reliable sources and was perceived as trustworthy.

Three different participants stated:

… if the guidelines seem reasonable to you, you wouldn’t think it’s required or

necessary to look into the research behind it.

Chapter 3 76

… the only reason [to consult supporting research] would be if I were to be held

responsible for some decisions being made and then if it didn’t work out I could

go back and say, “well I did this because of this really important…” That’s what

research is for, to reduce blame on yourself.

… The sources are in there and the guidelines come from a reliable source, so I’m

not going to look at them.

It is not clear why participants believed the research sources to be reliable. But both focus groups’ participants were clear that a combination of the time constraint any design project imposes coupled with their notion that the guidelines were common sense obviated further investigation on their part; “if you look at it [the guideline] and it seems reasonable, it’s already done its job. Looking into further research is just not worth the time.”

Perceiving guidelines as “common sense” does not imply that they were considered redundant or without value. In many cases, the guidelines reinforced existing knowledge.

Participants suggested that the guidelines could be best used as a checklist which would, “remind you of things you might have forgotten, not much [for] introducing it to you for the first time.”

Other participants did not differentiate between previously known or new design ideas, but rather suggested using the guidelines as a memory aid, “… a checklist and go through each guideline and see if you’ve implemented the guideline and if you have haven’t then you might consider implementing it or not.”

Chapter 3 77

Guidelines could also help solve design problems which involved trade-offs. Trade-offs, in a design context, refers to two or more mutually exclusive, yet valid, solutions to a problem.

“If a trade-off had to be made,” said one participant, “like if something had to be a big word but might create clutter somewhere else, you can weigh one guideline over another and figure out which is more important to do.”

Various participants spoke to the use of guidelines not at the start of a design exercise, but rather once a prototype had been developed, “I think I would design it first the way I think it should be and how I think other people would know how to use it, then I would look at the guidelines and see which guidelines could make this improved or better.” Guidelines which were not implemented at the prototype stage could then be explored, which – as one participant suggested – would be particularly helpful in cases where a new design feature was being considered. In both focus group sessions, participants said that the consideration of guidelines would occur only after the design was almost complete in order to retroactively find support for the design choices made:

[Participant E] The only reason that I would actually use these guidelines in the

future, if I wasn’t in school, is to justify anything I did design. I would design it

first, and if I had to justify certain aspects of whatever I make, I would go to the

guideline and this is what it says and…

(laughter) … because someone else said so…

… that’s how I would honestly use the guideline.

Chapter 3 78

[Participant F] You had to cover your butt for any reason, the guidelines would

be one way.

[Participant G] It’s because we kind of know the guidelines, so we would just

use that to backup.

As seen in the words of the participant who discussed her rationale for not consulting the

guideline sources, the potential use of the guidelines and their role in the design was about

mitigating blame. “That’s what research is for,” said the participant, “to reduce blame on

yourself.” While this attitude might suggest a dismissive attitude regarding the guideline set, in

fact the participants viewed the set as an authoritative and useful tool. They trusted the guidelines

for two reasons; because the guidelines were perceived to have come “from a reliable source”,

and – most importantly for the participants – the guidelines reflected existing knowledge or

opinion,

… everyone has an idea of what a good web-site is. Most of us have a few web

sites in our mind that we consider well designed and when we do look at these

guidelines we think back and reflect on these web pages and think, oh yeah, this

does occur pretty often on the good or better web pages. That way we can decide

whether or not a guideline is good.

Chapter 3 79

Knowledge of a “good web site” also contributed to whether participants thought a guideline set, or even a particular guideline, was valid. Participants were asked what criteria they might apply in deciding to consult a particular guideline set. One participant suggested the need for a link to exist between their prior knowledge of what constitutes a “good” site and the advice provided by the guideline set. Other participants noted that the use of a guideline was influenced partially by their familiarity with the guideline concepts, or whether the guidelines were, to use a common term in the focus group sessions, “common sense”. If the guideline ideas resonated with the participant, they would be more likely to use the guideline in their design work.

Asked if there were ways the guidelines could be improved, participants offered various solutions. Some liked the print version of the study booklet, since it allowed them to have the guidelines next to their computer while working, and because they, “don’t like staring at a screen for too long.” As reflected in the quotes below, other participants would have preferred an online version, one that would take advantage of hyperlinks and potential access to more examples:

It would be easier if it were available online, if you could access it from

anywhere. If you working on a website you could go online to the same place

every time instead of carrying it around with you.

I would prefer online over using it in book format.

It would be good to have online as a resource, but personally I prefer the booklet

and the ability to look through it.

Chapter 3 80

Given the critique of the guidelines, there were positive comments beyond the potential usefulness of the guidelines as a checklist, memory tool or as a supporting document for a design decision. Specifically, the layout of the guidelines was praised. One participant’s comments best sum up the opinions from both focus group sessions regarding the presentation of the guidelines in the guideline booklet;

I kind of like the way the pages were laid out too. They were consistent all the

way through. They had a diagram to demonstrate what a website should look like

because of this guideline or what it shouldn’t look like because of the guideline

and I thought the blurb was concise and to the point. If you want further

background information, the sources are listed. So if you need to do further

research or further digging, you could just check the source.

Both group interview sessions concluded with a discussion of the importance of context; that the use of specific guidelines depended on aspects of the design project. Regardless, it was suggested that guidelines could not address all aspects of design, but rather were a starting point towards finding design solutions. As one participant stated in conclusion,

I think overall the guidelines in question and guidelines overall are useful. They

should be part of the design process, but not necessarily exhaustive of your entire

process. They are avenues you should also explore and maybe the guideline is a

starting point, but there are other avenues you should explore when designing

Chapter 3 81

like checking out other websites, doing a literature review, or doing usability

testing iteratively.

This was a sentiment shared by other participants. Guidelines were a potential tool, an aid that

should be included in the mix of information requirements of a designer. But when guidelines

should be consulted depended on both the context of the particular design process and time

constraints.

3.4 Summary

The study revealed a variety of perspectives regarding the potential use of guidelines in the design process; the role of familiarity or “common sense” in whether or not a guideline should be applied to a design problem, problems regarding the interpretation of guidelines, preferences on the medium in which guidelines are presented, and various opinions regarding when and how guidelines should be used in the design process. These issues have been described to various degrees in previous studies (Kim, 2010; Ivory & Megraw, 2005; Tetzlaff & Schwartz, 1991), and the findings in this study support previous research. While the study is limited by the small number of participants (16) and the context of their work (the technical constraints of a Wiki page within the boundaries of a course assignment), the data contained within the booklets and focus group session transcriptions begin to address research questions posed in this dissertation.

The idea that guidelines might best be incorporated into the design process in the form of a checklist was noteworthy for a number of reasons. Checklists differ from guidelines in terms of brevity, but also in how the participants thought that they could be used – at the closure of the design phase, as opposed to points throughout. Checklists are arguably a different means to

Chapter 3 82

codify knowledge, although if participants reported problems interpreting guidelines, what

potential issues regarding interpretation would emerge from increased brevity? Would brevity

provide the specificity participants requested in the focus group sessions? The examples

provided in the guidelines were not deemed helpful and the cited research in the guidelines was

not consulted, which suggests that a checklist which does not include examples or cited research might find acceptance by designers. The use of checklists in the fields of clinical medicine

(Gawande, 2010; Pronovost , Needham, Berenholtz et al., 2006) and aviation (U.S. Federal

Aviation Administration, 2011) suggests a format that might also benefit design practice.

The use of guidelines as a means to support design solutions after they have been implemented was unexpected. Referencing guidelines once a project is complete reflects the attitude of participants regarding guideline sets and their own expert opinion. Expertise comes first, and the use of guideline sets follows. Why this attitude exists is unclear. Was there something about this particular participant group that led to this approach, or does this represent a broader attitude towards guideline use?

While some participants suggested only using the guideline set in support of design decisions previously made, this opinion was not a consensus. The focus group sessions suggested the use of guidelines at various points in a design project (prior to design, at some point in the middle, or upon conclusion). This variation in points of use may have had to do with the practice of specific designers (how they approach a design project), as well as their familiarity with guidelines and design experience. But this can only be surmised.

The use of the guidelines also brought to light problems regarding level of specificity.

How specific a guideline should be is very difficult to answer. Can guidelines be written representing broad research findings, but still include specific examples? While a guideline such

Chapter 3 83 as Guideline 14:11 (Use a graphic format to display data) was found by participants to be unclear and too broad, consideration of this specific design issue by designers during the design process may result in better design because the issue was raised. Similarly, the amount of white space, as discussed in Guideline 6:11, might be unclear and be context dependant, but arguably participants are now aware of the guideline and may reflect differently on their design. Ideally guidelines are not intended only for reflection on a design problem, but to offer some potential solution. For this reason, Guideline 6:11 (Limit the amount of white space) is particularly problematic: not only was the solution unclear, but participants were not certain that it addressed an actual problem. As one participant noted, the guideline was “bizarre”, a strong indictment of a tool intended to aid the designer.

Guidelines 6:4, 6:7, 6:11, 11:7, and 14:11 were all considered difficult to interpret.

Problems with interpretation does not mean that the guideline was not used, but only that the ideas present in the guideline were unclear. For example, Guidelines 6:7 and 11:7 were applied by all 16 participants. The problem is not whether they were applied, but rather how they were understood in their application to a design problem. If these guidelines were difficult to interpret, and if this difficulty led to variations regarding the interpretation, then the design solutions might be flawed – or rather, might not be the solution intended by the writers of the guidelines.

The Research-based Web Design & Usability Guidelines set is the strongest set of research based guidelines currently available to designers, and the subset used in this study had the highest levels of research support and included both illustrations and textual examples. Yet interpretation remained a concern. Guideline 14:11 (regarding the use of graphic formats in the display of data) was also found to be problematic; the description, illustration and examples included with the guideline were not enough for the designers to be certain of how the guideline

Chapter 3 84 should be applied. Although each guideline included cited research, the task of finding the supporting literature was deemed to be too time consuming by the participants. None of the 16 participants consulted the literature given time demands, demands which are present in any design project.

This study did not involve designers, but rather students of design and usability who were tasked to design web pages in a limited design environment. The use of a Wiki site suited the purposes of the class, and also arguably reflected real world technical design limitations – there are always constraints in a design project. Would designers make better use of guidelines if they were not limited by a Wiki environment? The answer is less important than a consideration of how guidelines were used, given potential restrictions to design. While the present study had limitations both in terms of the participants and the tasks involved, the data leads to a better understanding of a particular group’s opinions of design guidelines, how a specific evidence‐based web design guideline set is interpreted by those involved in the design process, and when guidelines might be used in the design process to solve design problems.

The participants used a combination of experiential knowledge and the searching out of similar sites (examples) in order to design. It is worth considering whether that approach represents the norm in the design of interfaces. The study found that guidelines play a marginal role in the design process. While the student participants had been previously exposed to guideline sets, they did not explicitly use those guideline sets in the design of their project. They only used the guideline set provided by this study and internalized what they had been taught to address design problems. Nonetheless, their desire to keep the booklet following the study speaks to their recognition that the guidelines were useful, even if there were problems with clarity and interpretation.

Chapter 3 85

The findings from Study #1 led to the design of subsequent studies. Study #2 focused on the third research question (RQ3), “How are evidence‐based web design guidelines interpreted by web interface designers?” Instead of engineering students, this study involved the participation of professional designers. Might their experience, their notion of “common sense”, differ from the participants of this first study such that professionals using the guidelines would not have the same problems regarding interpretation?

Chapter 3 86

Chapter 4

The interpretation and use of web interface design guidelines for website assessment (Study #2)

4.1 Introduction

Design guidelines involve both the interpretation of design knowledge in the crafting of a guideline and the interpretation of the guideline into design practice. Any representation of knowledge has the potential for misinterpretation and guidelines are no exception. Research has reported problems with both the understanding of guidelines by designers and with the misapplication of guidelines (Ivory & Megraw, 2005; Ivory, Sinha, & Hearst, 2001;

Vanderdonckt et al., 2000; Tetzlaff & Schwartz, 1991; Kabir, 2008). This study investigates disagreements between judgments of guideline adherence made by designers and considers how guidelines are interpreted. The study addressed the following research question; RQ3. How are evidence‐based web design guidelines interpreted by web interface designers?

4.2 Methodology

The study asked eight web interface designers to assess ten web sites using a set of guidelines.

After completing the assessments, the participants were paired and their discussion of assessments on which they disagreed was analyzed in order to better understand how the guidelines were interpreted and why there was disagreement regarding the assessments.

4.2.1 Method

Using 67 guidelines drawn from the Research-based Web Design & Usability Guidelines (U.S.

Department of Health and Human Services, 2006), eight web interface designers were asked to

Chapter 4 87

independently score ten cancer health information sites. In the first phase of the assessment

process, the participants completed their individual assessments over a period of one or two days.

Each participant made 670 assessments (67 guidelines x 10 cancer health information sites)

judging whether a site complied or did not comply with a specific guideline, or whether the

guideline was not applicable. There were no criteria outside of the guidelines themselves for

informing the assessments made by the participants, except that the choices were to be

considered mutually exclusive and that all ten sites had to be assessed against all 67 guidelines.

In addition, participants were asked to make brief notes regarding their assessments which could be used as a memory aid during future discussions. Participants used their own computers to access the 10 sites, which were all available in a captured format at a specified URL.

Once the individual assessments were completed, the second phase of the assessment process paired the eight participants to discuss their evaluations with the researcher. The purpose of this meeting was to discuss any disagreements and attempt to reach consensus. These meetings were moderated by the researcher and audio recorded.

Prior to discussing assessments where there was disagreement, the researcher randomly selected five guidelines for each participant pair where they had agreed. The interpretation of these guidelines was discussed to ensure that the two participants had interpreted and applied the guideline in the same manner. In all cases, the participants interpreted the guidelines consistently. Apart from the randomly selected guidelines, other guidelines on which the participants agreed during the individual assessment phase were not considered, since such agreement suggested that the participants agreed in the interpretation and application of a guideline. While recognizing that five randomly selected guidelines represent a fraction of the

670 assessments, potential participant fatigue prevented the discussion of all assessments.

Chapter 4 88

Three of the paired meetings took place in Toronto and one in Montreal. All the meetings

were conducted in person. The shortest paired meeting lasted over four hours, whereas the

longest meeting was conducted on two occasions totaling over seven hours.

4.2.2. Apparatus

The study used a version of the Research-based Web Design & Usability Guidelines (U.S

Department of Health and Human Services, 2006). This set was abbreviated by two factors: only

guidelines which were supported by moderate or strong research support were included and

guidelines which considered the design process were excluded, since any process employed was

unknown to the assessors. Of the 209 guidelines in the publication, 80 were supported by either moderate or strong research support (represented by four or five bullets on the strength of evidence scale). Thirteen of these guidelines were excluded because they focused on the design process. This left a total of 67 guidelines (see Appendix E).

The ten cancer health information sites were selected using comScore Media Metrix, a proprietary online traffic measurement tool which uses samples from over 10,000 Canadian households to rank websites in various categories. The study used the ten most visited (highest traffic over a period of one month) health information sites with specific information on cancer.

The ten sites with traffic levels (measured as total unique visitors per the month of October 2007) are shown in Table 4.1.

Chapter 4 89

Table 4.1: Cancer Health Information websites ranked by traffic levels (measured as total unique visitors in October 2007)

Rank Site Total Unique Visitors

1 NIH.gov 1,393,000 2 About.com Health 1,012,000 3 Medicinenet.com 617,000 4 Mayoclinic.com 604,000 5 Medhelp.org 307,000 6 Revolutionhealth.com 264,000 7 Cancer.ca 214,000 8 iVillagehealth.com 211,000 9 Netdoctor.co.uk 188,000 10 Everydayhealth.com 140,000

The assessments were not conducted on live versions of the sites, but rather using screen

captured versions, a process involving copying the existing web sites and transferring them to

another web server, thus preserving the sites in a consistent state. This ensured the assessors were all viewing identical sites.

For the assessment, participants were provided with the abbreviated guideline set in both print and electronic format (Adobe Acrobat PDF format) and received a paper checklist listing the guideline identification numbers, guideline titles, check boxes for MEETS, DOES NOT MEET and

NOT APPLICABLE and a space for notes (see Appendix F). This checklist was adapted from an

assessment aid used in a similar study (Cherry, Muter & Szigeti, 2006). The modified tool was tested by three volunteers to ensure usability.

Chapter 4 90

4.2.3 Participants

The participant designers all had at least six years experience designing and developing website

interfaces and had played a lead role as designers on both smaller (budget of less than $20,000)

and larger (budget of over $500,000) projects. Three of the eight participants were women and

two of the participants were located in Montreal, with the remainder in Toronto.1

4.2.4 Procedure

The individual evaluations were completed by the participants in isolation from the other participants. Participants who knew each other were also instructed not to discuss the study. Each participant was asked to use the assessment tools (comprised of both the guideline set and the checklist, described above in Section 4.2.2), and use captured versions of the health information websites to conduct the assessments. Once the individual assessments were completed, the participants were paired by geographic expedience; the two participants in Montreal were paired for convenient geographic reasons, whereas the remaining six participants were paired by geographic proximity within the city of Toronto. Pairings based on proximity were necessary due to the work schedules of the participants and the time requirements of the study (since meetings were correctly estimated to take at least three hours). These meetings were held in various locations of convenience to the participants, where privacy was assured.

4.2.5 Measures

There are three primary variables in the study: the 67 guidelines, the assessments (individual and paired) and the ten cancer health information websites. The study generated two data sets, the

1 The study protocol was approved by the Ethics Review Office of the University of Toronto. Participants were compensated by receiving a copy of the Research‐based Web Design & Usability Guidelines (U.S. Department of Health and Human Services, 2006).

Chapter 4 91

first with 5360 data points (8 participants making 67 assessments across 10 web sites) and the

second with 2680 data points (4 paired participants agreeing on 67 assessments across 10 web sites). In addition, the recordings of the discussions shed light on the process of interpretation, as participants discussed their rationale for their assessments, focusing on assessments where there was disagreement between participants. Transcriptions of the meetings were coded and analyzed.

4.3 Results

Data was comprised of both the assessments of the cancer health information websites using the 67 guidelines, and the discussion of the assessment on which there was disagreement by the paired participants. The data will be discussed in a similar order; first the assessments of the websites, and then the related discussion of the guidelines for which there was disagreement by the participants.

4.3.1 Assessment of cancer health information websites

The individual assessments revealed guideline compliance across the ten websites varying from

25 to 57 guidelines (out of 67 guidelines). Table 4.2 lists the ten sites, and the number of guidelines with which the site complied, by each of the participant assessors.

Chapter 4 92

Table 4.2: Assessments by individual participants. Sites ranked by the average number of guidelines with which they complied (maximum possible was 67).

Participant designers2 Rank Site A B C D E F G H Average

1 Mayoclinic.com 54 57 45 47 54 54 39 45 49.4 2 Cancer.ca 46 49 53 49 48 50 40 48 47.9 3 Revolutionhealth.com 48 51 53 49 43 45 44 44 47.1 4 iVillagehealth.com 51 40 44 45 51 51 32 43 44.6 5 Cancer.gov 44 40 49 40 51 40 37 48 43.6 6 Everydayhealth.com 39 36 42 47 40 52 44 44 43.0 7 Medicinenet.com 34 55 32 51 50 46 27 43 42.3 8 Netdoctor.co.uk 38 34 37 45 43 54 37 41 41.1 9 Medhelp.org 34 50 28 37 44 45 43 47 41.0 10 About.com Health 26 27 38 47 41 36 25 40 35.0

Across the ten sites, the average compliance ranged from 35 (52.2%) to 49.4 (73.7%), where

mayoclinic.com complied with the largest number of guidelines on average, and

about.com/Health complied with the lowest. The lowest rated site complied with an average of

slightly more than half the guidelines (35 of 67 guidelines, or 52%).

Following the completion of individual assessments, participants were paired and met to

discuss assessments on which there was disagreement. Each of the four pairs reached consensus on

all 670 assessments. Table 4.3 lists the ten web sites ranked by the average number of guidelines

with which compliance was noted by the four participant pairs. The average compliance ranged

from 33.5 (50.0%) to 49.5 (73.9%) across the ten sites, where cancer.ca the highest number of

guidelines on average, and about.com/Health complied with the lowest.

2 The eight individual participants are identified with single upper case letters. LATER, the paired participants are identified with double upper case letters, representing the pairing (AB represents the pairing of participant A and participant B).

Chapter 4 93

Table 4.3: Assessments by pairs of participants. Sites ranked by the average number of guidelines with which they complied (maximum possible was 67).

Paired participants Rank Site AB CD EF GH Average

1 Cancer.ca 50 52 50 46 49.5 2 Mayoclinic.com 56 49 44 46 48.8 3 Revolutionhealth.com 49 55 43 43 47.5 4 Cancer.gov 42 45 40 44 42.8 5 Everydayhealth.com 35 44 45 45 42.3 6 iVillagehealth.com 46 46 43 32 41.8 7 Netdoctor.co.uk 41 40 43 42 41.5 8 Medicinet.com 43 50 28 34 38.8 9 Medhelp.org 40 38 24 44 36.5 10 about.com/Health 24 40 39 31 33.5

The ranking of the sites based on the average number of guidelines with which they complied did not change greatly from what was observed in the individual assessments (as shown in Table

4.2), with three sites not changing ranked position, six sites moving either up or down one spot and only iVillageHealth.com moving two positions.

Table 4.4 combines data from Table 4.2 and 4.3 in order to present site rankings based on individual assessments, paired assessments and traffic.

Chapter 4 94

Table 4.4: Rankings of sites by individual participant assessment, paired participant assessment and traffic levels as per comScore Media Metrix data (October 2007).

Ranking Individual participant Paired assessment: Sites ranked by traffic levels assessment: Sites ranked by the average number of (total unique visitors) Sites ranked by the average number guidelines with which they complied of guidelines with which they (maximum was 67) complied (maximum was 67) 1 Mayoclinic.com Cancer.ca Cancer.gov 2 Cancer.ca Mayoclinic.com About.com Health 3 Revolutionhealth.com Revolutionhealth.com Medicinenet.com 4 iVillagehealth.com Cancer.gov Mayoclinic.com 5 Cancer.gov Everydayhealth.com Medhelp.org 6 Everydayhealth.com iVillagehealth.com Revolutionhealth.com 7 Medicinenet.com Netdoctor.co.uk Cancer.ca 8 Netdoctor.co.uk Medicinet.com iVillagehealth.com 9 Medhelp.org Medhelp.org Netdoctor.co.uk 10 About.com Health About.com/Health Everydayhealth.com

While there is some variation between how the sites rank when comparing assessments by individual participants and those following the discussions of paired participants, the greatest variation is between these two rankings (column 2 and 3) and the sites ranked by traffic levels

(column 4). The difference between sites ranked by guideline compliance and ranked by unique visitors suggests that the relationship between guideline compliance and web site use remains unclear. However, it is important to note that traffic level cannot be considered an indicator of user preference, since awareness of all ten sites by potential users is unknown.

Chapter 4 95

4.3.2 Website assessments

The site assessments were completed by the participants and submitted to the researcher, who

identified disagreements between subsequently paired participants. The study considered the

interpretation of the guidelines by (i) looking at the agreement rate between pairs of participants,

(ii) comparing the assessments of all four participant pairs to each other, and (iii) analyzing the

discussion of the paired participants as they worked to reach consensus on their assessments.

The study hypothesizes that if the guidelines are interpreted in the same manner, there

should not be any disagreement between assessments. Arguably, if two participants share an

identical understanding of a guideline, then their assessment should also be identical.

Disagreements in assessments indicate variations in interpretation. Table 4.5 illustrates the

number of disagreements between two (paired) individuals by website. The assessments of one

participant were compared to that of their participant pair – any disagreements were counted.

Disagreements represent different assessments (MEET versus DOES NOT MEET, MEET versus DOES

NOT APPLY, or DOES NOT MEET versus DOES NOT APPLY).

Chapter 4 96

Table 4.5: Number of disagreements within pairs of participants (prior to discussion)

Pair AB Pair CD Pair EF Pair GH Cancer.ca 18 14 24 15 Cancer.gov 21 15 24 23 Everydayhealth.com 22 23 25 16 Mayoclinic.com 12 26 13 16 Revolutionhealth.com 21 12 26 22 Netdoctor.co.uk 29 24 17 18 Medicine.com 28 30 27 30 iVillagehealth.com 28 27 19 32 About.com/Health 23 29 15 26 Medhelp.org 32 31 30 18 TOTAL 234 231 220 216

The number of disagreements within pairs ranged from 216 (Pair GH) and 234 (Pair AB). These disagreements were the focus of the discussions, which saw each pair meet with the researcher

(who moderated the meetings) in order to discuss the rationale for their assessments. Both participants discussed the assessments in light of both their interpretation of the guideline and in consideration of the specific cancer health information website. In all four pairs, the participants finished the meeting achieving consensus on the assessments across all the guidelines and all websites. In reaching consensus, changes to the original assessments could show bias towards one of the two individuals due to a number of factors, including the perceived knowledge of the other participant, the ability to debate a contentious point or by the presence of a dominant personality. These factors do not appear to have influenced any of the pairs. In one pair, the two designers each changed their assessments exactly 50% of the time (across 216 assessments). The other pairs of participants changed their original assessments in 41.6% and 58.4%, 40.2% and

59.8%, and 35.5% and 64.5% of the instances.

Chapter 4 97

In reaching consensus, participants combined to revise between 216 (Pair GH) and 234

(Pair AB) of the 670 initial assessments. Pairs AB and CD resolved 12 differences in their

assessments of mayoclinic.com and revolutionhealth.com respectively, the lowest number of

disagreements for a website. In terms of the greatest number of disagreements between a pair in

their assessments, Pairs AB and GH made 32 changes to the original individual assessment of

medhelp.org and iVillagehealth.com respectively (almost half of the total number of guidelines.)

These changes represent both disagreements on which consensus was reached following discussion, as well as guidelines where discussion led to a re-evaluation of both participants’ initial assessments. In the former case, one of the participants changed their assessment to concur with the other participant. In the latter case, both participants changed their assessments. This re-

evaluation of the original assessments by both participants was not expected. As there were three

options for assessment (MEET, DOES NOT MEET, or DOES NOT APPLY), there were cases where,

following discussion, two participants agreed on the third option, which neither had selected

prior to meeting.

The assessments revealed differences across the guidelines. Following the discussion

within pairs, there was agreement across all ten websites by all four pairs of participants on only

two of the 67 guidelines (representing 3% of the total.) In other words, there was consensus by

all participant pairs regarding only two of the guidelines; 8:1 (Eliminate horizontal scrolling) and

11:7 (Use familiar fonts). In some cases three of the pairs would agree on an assessment, while the other pair interpreted the guideline differently. Table 4.6 lists all the guidelines used in the study and the number of sites for which there was agreement in the assessments across the four pairs.

Chapter 4 98

Table 4.6: Sixty‐seven guidelines and the number of sites for which four pairs agreed

Category Guide Description Number of sites on which four pairs agreed line # 1 2 3 4 5 6 7 8 9 10 Optimizing the 2:3 Standardize Task Sequences User Experience 2:5 Design For Working Memory Limitations 2:6 Minimize Page Download Time 2:10 Provide Feedback When Users Must Wait 2:13 Do Not Require Users to Multitask While Reading Accessibility 3:3 Do Not Use Color Alone to Convey Information The Homepage 5:3 Create a Positive First Impression of Your Site 5:6 Ensure the Homepage Looks like a Homepage Page Layout 6:2 Place Important Items Consistently 6:3 Place Important Items at Top Center 6:4 Structure for Easy Comparison 6:7 Align Items on a Page 6:9 Avoid Scroll Stoppers 6:11 Use Moderate White Space 6:12 Choose Appropriate Line Lengths 6:13 Use Frames When Functions Must Remain Accessible Navigation 7:8 Keep Navigation‐only Pages Short 7:9 Use Appropriate Menu Types 7:10 Use Site Maps Scrolling and 8:1 Eliminate Horizontal Scrolling Paging 8:2 Facilitate Rapid Scrolling While Reading 8:3 Use Scrolling Pages for Reading Comprehension 8:4 Use Paging Rather Than Scrolling Headings, Titles 9:1 Use Clear Category Labels and Labels 9:3 Use Descriptive Headings Liberally Links 10:1 Use Meaningful Link labels 10:3 Match Link Names with Their Destination Pages 10:5 Repeat Important Links 10:6 Use Text for Links 10:9 Ensure that Embedded Links are Descriptive Text Appearance 11:1 Use Black Text on Plain, High‐Contrast Backgrounds 11:4 Ensure Visual Consistency 11:6 Use Attention‐Attracting Features when Appropriate 11:7 Use Familiar Fonts 11:8 Use at Least 12‐Point Font 11:9 Color‐Coding and Instructions 11:10 Emphasize Importance

Chapter 4 99

Category Guide Description Number of sites on which four pairs agreed line # 1 2 3 4 5 6 7 8 9 10 12:1 Order Elements to Maximize User Performance 12:2 Place Important Items at Top of the List 12:3 Format Lists to Ease Scanning 12:4 Display Related Items in Lists 12:5 Introduce Each List 12:8 Use Appropriate List Style Screen Based 13:13 Use a Single Data Entry Method Controls 13:22 Use Data Entry Fields to Speed Performance (Widgets) 13:25 Minimize Use of the Shift Key Graphics, Images 14:1 Use Simple Background Images and Multimedia 14:2 Label Clickable Images 14:3 Ensure that Images Do Not Slow Downloads 14:4 Use Video, Animation, and Audio Meaningfully 14:5 Include Logos 14:6 Graphics Should Not Look Like Banner Ads 14:10 Include Actual Data with Data Graphics 14:11 Display Monitoring Information Graphically 14:13 Emulate Real‐World Objects 14:15 Use Images to Facilitate Learning Writing Web 15:1 Make Action Sequences Clear Content 15:2 Avoid Jargon 15:6 Use Mixed Case with Prose 15:7 Limit the Number of Words and Sentences 15:9 Use Active Voice 15:11 Make First Sentences Descriptive Content 16:1 Organize Information Clearly Organization 16:2 Facilitate Scanning 16:4 Group Related Elements 16:7 Display Only Necessary Information 16:9 Use Color for Grouping

If even a single pair of participants reached a different assessment of a site from the other three pairs, the site assessment was not recorded in Table 4.6 as an agreement. Table 4.6 reveals a high degree of disagreement across the pairs. For 65 of the 67 guidelines there was at least one site for which pairs differed in their assessments. Further, the pairs did not agree on a single website in their assessment of eight of the guidelines (2:3, 2:5, 2:6, 7:9, 14:2, 14:3, 14:4, and 14:13). Six guidelines (6:3, 8:4, 12:2, 10:9, 13:13, and 13:22) found consensus on only one of the ten sites.

Disagreement suggests differences in guideline interpretation.

Were there specific characteristics of guidelines for which there were high agreement rates versus those with low agreement rates that would suggest ways to improve guidelines?

Chapter 4 100

Such a comparison of characteristics of different guidelines might reveal aspects which could lead to the improved interpretation of a guideline.

4.3.3 Characteristics of the sixty-seven guidelines

Guideline characteristics were assessed along six dimensions:

1. The total number of sources cited in support of the guideline.

2. The total number of sources cited since 2000. This dimension was included in order to

explore whether the recency of sources influenced interpretation of a guideline.

3. The total word count for the description of the guideline.

4. The total world count for the comments associated with the guideline.

5. The number of examples provided with the guideline

6. The tone of the guideline. This final dimension was included to explore differences

between guidelines which instructed the user to take action (i.e., Use Familiar Fonts) or

instructed them not to take action (i.e., Eliminate Horizontal Scrolling)

The agreement rate represents the number (from zero to 10) of websites for which all individual participants agreed in their evaluation. Regression and correlation analyses were conducted to investigate the relationship of any of the six dimensions with the agreement rate of the guideline.

The results showed that none of the dimensions correlated significantly with the agreement rates of the guidelines (the largest correlation was -.134 between the word count in the comments associated with the guidelines and the agreement rate of the guideline) and there was no significant predictive relationship of any combination of the dimensions on the agreement rate of the guideline.

Chapter 4 101

Cluster analyses were carried out to see if non-linear combinations of dimensions might

define clusters of guidelines that differed in terms of how relevant they were to the websites that

were studied. Both partitioning (k-means analysis) and hierarchical cluster analyses were used

(as implemented by the SPSS statistical package). A number of different clustering solutions

were identified, but no clusters were consistently found across the different clustering

techniques. Furthermore, the various clusterings obtained did not differ in terms of the average

agreement rate of the guidelines within each cluster. The clustering solution obtained with

Ward’s method (using standardized versions of the dimensions) is shown below (Figure 4.1.)

Note that while the dendrogram shows a structural relationship between the guidelines, there

appears to be no relationship between differences in clusters and differences in guideline agreement rates.

Chapter 4 102

Guidelines (Abbreviated Title and Number)

Figure 4.1 Dendrogram using Ward Linkage

Chapter 4 103

Nine percent of the guidelines were applied consistently to either nine or ten of the cancer

health information sites. This suggests that these guidelines were interpreted in a similar manner.

In contrast, 12% of the guidelines did not lead to agreement by all four pairs on a single web site.

Table 4.7 lists the six guidelines with the highest rates of agreement across the four paired

assessments (two guidelines had unanimous agreement across the ten sites by all four pairs of

participants and four guidelines had agreement across nine sites). Agreement represents a guideline where all four pairs of assessors submitted an identical assessment.

Table 4.7: Characteristics of guidelines with high agreement rates

Guideline: 8:1 Eliminate 11:7 Use 6:7 Align 6:12 Choose 14:11 Display 15:6 Horizontal Familiar items on appropriate monitoring Use mixed Scrolling Fonts a page line lengths information case with graphically prose Agreement 10 10 9 9 9 9 Rate Number 5 sources: 2000 8 sources: 13 sources: 12 sources: 5 sources: 12 sources: and year (x2), 2001, 2002 1995, 1998 1968, 1984, 1929, 1940, 1961, 1981 1928, 1944, of sources (x2) (x2), 2000 (x2), 1986 (x2), 1979, 1983, (x2), 1986, 1955, 1963, 2001 (x2), 2002 1994, 1995, 1986, 1988, 1994 1968, 1971, 1996 (x3), 1998 1998 (x2), 1975, 1977, (x2), 2000 (x2) 1999, 2000, 1984, 1986, 2001, 2002 1987, 2000 Word count of 14 11 8 32 13 10 guideline Word count of 40 28 26 99 47 67 commentary Number of 2 1 3 2 2 1 examples Tone negative positive positive positive positive positive

The average number of sources (cited research) for these guidelines was nine. Of the 55 sources

referenced by the six guidelines, 16 (28%) were published in 2000 or later. The average word

Chapter 4 104

count of the guideline description was 15 and the average number of words used for commentary

was 51. Each guideline was further supported by almost two examples on average.

The language employed in the guidelines was also considered, since the words used to

both define the guideline and used in the additional commentary section might influence the

interpretation of the guideline. Guideline 8:1 has a negative title (“Eliminate horizontal

scrolling”), but a positive description (“ Use an appropriate page layout to eliminate the need for

users to scroll horizontally”), which is why the tone was recorded as a “mix” in the table.

Guideline 6:7 uses positive language in both the title (“Align items on a page”) and the

description (“Visually align page elements, either vertically or horizontally”). By comparison,

Table 4.8 highlights the characteristics of guidelines with no agreement.

Table 4.8: Characteristics of guidelines with no agreement

Guideline 2:3 2:5 2:6 7:9 14:2 14:3 14:4 14:13 Standardize Design for Minimize Use Label Ensure That Use Video, Emulate Task Working Page Appropriate Clickable Images Do Audio and Real World Sequences Memory Download Menu Types Images Not Slow Animation Objects Limitations Time Downloads Meaningfully Agreement Rate 0 0 0 0 0 0 0 0 # and year of 11 sources: 16 sources: 8 sources: 2 sources: 7 sources: 15 sources: 11 sources: 4 sources: sources 1985, 1986 1958, 1975 1983, 1997 1980, 2000 1931, 1957, 1984, 1986, 1993, 1995, 1989, 2000 (x2), 1987, (x2), 1987, (x2), 1998, 1975, 1997, 1995, 1996, 1997, 1999, (x2), 2002 1988, 1990, 1992, 1996, 1999 (x2), 1998 (x2), 1997 (x2), 2000 (x5), 1996, 1997 1998 (x2), 2000, 2002 2000 1998, 1999, 2000/2001, (x2) 1999, 1999, 2000 (x3), 2001 2000 2000(x3), 2001 (x3), 2001 (x2), 2003 2002 (x2) Word count of 14 15 10 24 14 17 22 10 guideline word count of 45 122 17 99 44 153 83 92 commentary Number of 2 0 0 2 1 1 0 4 examples Tone positive negative negative negative positive negative positive positive

Chapter 4 105

The average number of sources for these guidelines was 11. Of the 74 sources referenced by the seven guidelines, 29 (39%) were published in 2000 or later. The average word count of the guideline description was 18 and the average number of words used for comments was 94. No examples were provided for three of the guidelines provided, two were supported by one example, two by two examples and one guideline had four examples. Half the guidelines employed negative language, while the other half used positive language.

Tables 4.7 and 4.8 represent guidelines on which there was the most and least agreement.

There are no notable differences between the characteristics of these two groups of guidelines.

The average word count and examples were almost identical. The guidelines for which there was the least agreement were supported by more recent research (39% of the supporting research was published in 2000 or later, compared to 28% of the supporting research for the guidelines with the highest agreement rates). Both groups of guidelines had evidence of ambiguous wording

(which can lead to uncertainty and variations in interpretation.) Based on these two tables, there are no characteristics of guidelines that can be said to differentiate their interpretation. Next, we consider guideline interpretation not through a look at the characteristics of guidelines, but by the discussion of guidelines by participants.

4.3.4 Participant discussion and the interpretation of guidelines

Participants assessed ten web sites against a set of 67 web design guidelines. As discussed in

Section 4.2.2., participants were provided with three possible choices in making their assessment: MEETS, DOES NOT MEET and NOT APPLICABLE. They were not given details regarding what constituted the three conditions; such instructions would potentially bias interpretation.

Following the independent assessments, participants were paired and asked to discuss the

Chapter 4 106

interpretations where there was a disagreement in their assessment of a website. The discussions

revealed five key problems involving the interpretation of guidelines. First, an uncertainty

regarding what constitutes adherence or non-adherence to a guideline: Inherent in the

interpretation of a guideline was an understanding of the degree to which the ideas captured by

the guideline were to be applied in a web-page design. Second, how a guideline is interpreted can

depend on a single word. The presence of ambiguous words in a guideline (such as

“appropriate”) results in greater uncertainty in the interpretation of a guideline. How a single

word is understood affects the interpretation of the guideline. Third, previous experiences of a

designer influence interpretation, particularly if the guideline recommends a course of action that

runs counter to design practices. Fourth, where participants found contradiction between

guidelines within the set, problems with interpretation were revealed. Finally, examples provided

by the guidelines should ideally have provided clarity, but the participants did not always find

this to be the case.

Uncertainty regarding adherence conditions

By asking participants to assess web pages against specific guidelines, the study implicitly raised

questions regarding the conditions which constituted adherence. When assessing a web page,

would a single example of guideline contradiction mean that the web page DOES NOT MEET a specific guideline? Or is the inverse correct and would a single example of adherence result in an assessment of MEETS? The paired discussions addressed these questions, as specific guidelines

were considered by the participants. In particular, discussion of guidelines 11:6 and 8:3 revealed the nature of this interpretation problem. The paired discussion of Guideline 11:6 (Use Attention-

Attracting Features when Appropriate) illustrated this;

Chapter 4 107

[Participant G] ... another one of those guidelines, where it’s pretty fuzzy as to how

much… how many attracting features are required to meet the guideline? Like, this page

has one feature, does that mean that the site meets the guideline? Overall, I would say

that it doesn’t meet because the text is all the same colour, all the same size for every

different section of the page, the text in the side bar is the same as the text in the main

area. Generally.

The participant described two potential answers to the question of the number of features

required for the web site to be considered in compliance with the guideline; a very objective

single instance or a more subjective view where the features described by the guideline need to

pass some threshold. This second interpretation suggests that if a web page “generally” meets the

guideline, then the page has adhered to the guideline. The two participants agreed on this

interpretation for this particular guideline, but not necessarily for other guidelines.

Descriptors used by the participants such as “overall” and “generally” suggest that participants, when viewing the web pages, employed some undefined threshold regarding the presence of elements described in the guideline. In the above example, Participant G felt that not enough attention attracting features were appropriately used since the font colour and sizes did not adequately differentiate sections of the page. The other participant in the discussion disagreed.

[Participant H] See, I would say yes [it meets the guideline]. The guideline talks about

“appropriate” amount of graphic to me, for a cancer site, this strikes me as the

Chapter 4 108

appropriate amount. If it were a kid site it might not, but there is enough difference in the

heading styles and font styles and the use of imagery…

[Participant G] Yeah. I guess it is appropriate for the subject matter. I can’t think of

anything terribly relevant outside of images of people, which looks nice but is not

necessarily helpful. So I agree.

Both participants in this example consider font sizes and colours as examples of attention attracting features. Where they differ is in how this guideline is interpreted in the context of a cancer health information site. Features which are appropriate to employ on one type of website

(such as a site aimed at children) differ from features appropriate for a website directed at a different audience (for example, a health information website.)

The contextual relevance of the content to this guideline was a factor in other discussions.

In these cases, the discussion regarding interpretation was very specific to a website. Guideline

11:6 (Use Attention-Attracting Features when Appropriate) recommends that websites, “use attention-attracting features with caution and only when they are highly relevant.” The question of relevance was important in the interpretation and determination of compliance conditions for the guideline. In one participant exchange, the debate surrounded the use of photographs as an attention attracting feature.

[Participant B]: I put MEETS because of photos of cancer.

[Participant A]: But the photos are irrelevant and don’t add anything.

Chapter 4 109

[Participant B]: I thought with the case of skin cancer, where you need to see things on

your skin, it was relevant.

[Participant A]: I saw that and didn’t think it was useful. The guideline said relevant.

Something totally relevant would be good, but it’s not here.

[Participant B]: I know across all the sites I would go looking for skin cancer pictures,

because if you were using pictures anywhere it would be these instances.

One participant made a compelling case regarding the relevance of photographs illustrating skin cancer (which is an image they looked for across all ten of the websites being assessed), which the other participant did not find relevant. Although the participants reached consensus regarding this guideline (the presence of skin cancer photographs was deemed to be highly relevant), their original positions were quite different.

Guideline 8:3 recommended that designers “use longer, scrolling pages when users are reading for comprehension.” Apart from the potential uncertainty of whether or not users are

“reading for comprehension”, the participants were unclear how many pages were required in order for them to declare guideline adherence;

[Participant D] In this instance, there were so many differences on pages – in some

instances they did follow the guideline, in other instances they did not.

Chapter 4 110

[Participant C] I agree. It depends what section you’re in.

[Participant D] The guideline says most of the time, so I guess it does meet by that

criterion.

In this example, “most of the time” was the phrase the participant referenced to reach agreement.

However, this is not a term found specifically in Guideline 8:3, implying that the interpretation of guidelines was influenced by readings of other guidelines (where the term can be found) in defining the criteria for assessment. Another participant pair considered the term “most” too ambiguous and they felt they had to draw a clearer line in order to be confident in their assessment. The discussion of what constitutes adherence returned to the question of whether a single example of either guideline violation or adherence would lead to a particular assessment.

[Participant G] I interpreted it that I couldn’t find an example where it violated the

guidelines. So I selected MEETS.

[Participant H] If I can’t see an example of it not complying, then I would be fine.

[Participant G] I can’t say for certain that on the whole site this guideline is violated, so

essentially I would answer as maybe. Could I find a case of violation of the guideline?

That was the question I asked.

Chapter 4 111

[Participant H] I took the opposite route – whether I could definitively find example of

the guideline being met.

[Participant G] If you wanted to rule out any instance of the guideline being violated,

you’d have to spend hours looking. Which isn’t that realistic.

Participant H ended the discussion of Guideline 8:3 with an important question, “Would one

example of non-compliance be enough?” The question was echoed in the comments of another

participant pair, who debated how best to interpret adherence of www.cancer.ca to Guideline

12:5 (“Introduce Each List”). The guideline was not met in all cases, as noted by Participant A,

“It’s not consistently met.” Participant B thought this was not indicative of non-compliance,

because the design was stronger in instances where the guideline was not met. “I thought it was fine,” said the participant, “for the times that it did meet.” This led the other participant to wonder what threshold for compliance was necessary in order for the site to be considered

compliant with a guideline. “So again,” replied the participant, “where is the balance? Where is

the tipping point where [an assessment] should be considered MEETS or NOT MEETS?” The

specific guideline set used in this study was developed for use by designers, usability specialists,

managers and researchers “involved in evaluating Web design” (U.S. Department of Health and

Human Services, 2006, p. xvii). The need to define conditions of adherence is necessary not only

in the evaluation process, but also in the design process. For example, a designer would have to

know roughly how many “attention attracting features” should be used if they intend to comply

with Guideline 11:6 (Use Attention-Attracting Features when Appropriate.)

Chapter 4 112

The threshold for compliance emerged in the discussion of Guideline 11:8 (“Use at least

a 12-point font (e.g., typeface) on all Web pages”) as well. Participant D initially believed the

site under discussion did not adhere to the guideline, but Participant C disagreed;

(Participant C) I said it does, because the main body area is 12 point. I took a screen

capture and checked.

(Participant D) I’m with [Participant C] then. It does use it on some sections of the page,

but I don’t know if the guideline says… [voice trails off]

Regarding Guideline 11:8, all four pairs ultimately chose to interpret the guideline so that the presence of any font, in the main body of text, which appeared to be less than 12 points in size

would result in a site’s non-adherence to the guideline. The presence of text smaller than 12

point outside of what participants interpreted as the “main” or primary area of text content did not represent a contradiction of the guideline. With this particular guideline the threshold for adherence was clear, even if the screen resolution posed a notable problem. Two of the

participant pairs defined an additional “aid” in making their assessments; the presence of a

widget on the website which allowed users to alter the size of the font. If such a widget was

present, the participants recorded web site adherence to the guideline. It is important to note that

the use of a widget allowing users to change the font size is not mentioned in Guideline 11:8.

This was an interpretation of the guideline which emerged from interacting with the websites

(this particular widget was present in a number of the websites). Not only did Guideline 11:8

require participants to negotiate an interpretation, but two pairs “created” an interpretation to use

Chapter 4 113

during assessments. One participant asked, “was there a way to change the font size?” and

answered that, “the way we interpret this guideline is that if there is a way to adjust the font size

on the page, the guideline is met.” The ability to change the font size as an indicator of adherence is also captured in the following exchange,

(Participant F) The left handed navigation font was smaller. It should be used across the

site.

(Participant E) If a site offered the ability to increase the font, I took that into account.

It’s something all sites should have, considering the content.

What state constituted adherence on a page remained debatable, and differed across different

examples. As one participant remarked, “Sometimes I saw the guidelines in a very narrow

, whereas other times I thought of it in broader terms.” Whether a guideline was

considered with a narrow or broad interpretation depended at times on a single word. Returning

to the discussion of Guideline 11:6 (Use Attention-Attracting Features when Appropriate), the

participants focused their interpretation on the word “appropriate.” Deciding how to assess a site

depended on the participant’s interpretation of this one word.

Importance of a key word or phrase in guideline interpretation

Participant pairs discussed the word “appropriate” in a number of instances, recognizing the

importance of this key word. It was a word that resulted in uncertain interpretations of Guideline

8:1 (Eliminate Horizontal Scrolling), which states that designers should, “Use an appropriate

Chapter 4 114

page layout to eliminate the need for users to scroll horizontally.” The comment section of the

guideline provides more detail, “Horizontal scrolling is a slow and tedious way to view an entire

screen. Common page layouts including fluid and left-justified may require some users to scroll

horizontally if their monitor resolution or size is smaller than that used by designers.” The

participants were uncertain how to define the word, “appropriate.”

[Participant C] ... the question is what is appropriate width [to] eliminate horizontal

scrolling?

[Participant D] I disagree with this guideline, or rather the example they had. A 640 by

480 screenshot, but I based my assessment on 800 by 600.

[Participant C] I was confused too. The examples indicate that everything should fit into

640 by 480. But the guideline does say appropriate layout, which I would argue is 1024.

In order to interpret this guideline, it was necessary to agree on a working definition of

“appropriate” in the context of screen resolution. What is the appropriate screen resolution for a browser? Even though the guideline provided an example of a 640 by 480 resolution, the participants did not consider this appropriate. Instead, both participants felt a higher resolution

(800 by 600 or 1024 by 768) was more representative of current monitor settings and based their assessment accordingly.

In the case of Guideline 10:5 (“Repeat Important Links”), guideline interpretation was influenced by how participants defined which links should be considered important. One

Chapter 4 115

participant said the guideline, “calls into question what is important. Who says what is important,

the designers or the users of the site? Is it all important?” The question asked by the participant

raises another complicated factor – not only is the context of the website important in the

interpretation of the guideline, but so is the perspective of the assessor or user of the guideline –

are they users, designers, or both? How they define themselves will inform which links they

consider important. Even recognizing the polysemous nature of words, interpreting Guideline

10:5 requires the user to (i) articulate their own perspective (user? designer? both? other?), (ii)

define the notion of “important” from a particular perspective, and (iii) make an assessment.

These actions also take place within the context of the specific site under consideration, as seen

in the discussion of Guideline 11:6.

Guideline 9:1 (Ensure that category labels, including links, clearly reflect the information

and items contained within the category”) further explains that, “category titles must be

understood by typical users.” The participants discussed not how to interpret a “typical user”,

but rather how to define a category. Regarding one particular web site, a participant felt that, “in some cases the category labels could be more clear.” The other participant in the discussion was less concerned with the clarity of the labels as they were with understanding what the guideline meant by “category”. This was the key word in this guideline. “I looked at it a different way,” replied the other participant. “What is a category? Is the parent category a category? What about

the sublinks? Are they refined categories?” First the participants had to define category, and then

they could assess the site as to adherence. This was an issue faced by another participant pair,

[Participant E] So you think there is no clear category?

Chapter 4 116

[Participant F] See all this... [pause as the participant points to section of the site]. You

have to go to C to find cancer.

In this example, the letter “C” represented a possible category label, but it was not initially clear to the participants. Whether or not the letter “C” was a category label influenced the assessment of the site against the guideline. How to define the category depended on the specific site,

[Participant F] To find something in here, you have to find a question or go by a

definition that you find on the site. It’s not really easy to find what they have on the site.

[Participant E] The way I look at it, is that one of the categories is the post. Everything

[the content] is in there now. They are all in there now. Another category is the journal

entries.

By viewing the site together, the participants negotiated the interpretation of “category”. The post section of the website was interpreted by the participants to represent a category, and journal entries represented another category under which content could be found. The discussion of this guideline ended with one participant declaring that it, “depends on how you look it. How you interpret it.” How the participants interpreted the word “category”, determines how they interpret the guideline.

Discussing Guideline 7:8 (“Keep Navigation-only Pages Short”) in the context of www.mayoclinic.com, one participant asked, “what constitutes short?” Another participant, again raising the issue of screen resolution, asked, “There is a problem when you test the sites on

Chapter 4 117 different screens. What you see is different between a large and very small screen.” In other words, what constitutes short on one screen may not be short on another.

There were also phrases that required clarification to allow for guideline interpretation.

When assessing whether a visual representation of a video player on mayoclinic.com complied with Guideline 14:13 (“Emulate Real World Objects”), one participant asked, “What is a real world object?” Another participant pair struggled with the same guideline and discussed whether buttons represented real world objects. “They’re not real world objects,” debated one participant.

When discussing www.medicine.net, participants were uncertain about the terms used in the title of Guideline 2:3 (“Standardize Task Sequences”),

[Participant E] What is standardized task sequence? There may be misunderstandings

with this guideline.

[Participant F] I focused on the fact that things didn’t change that much – was the

interface constant or not.

[Participant E] For me each section on this site had its own internal logic. When you

move around sections, this doesn’t exist. The problem is in the guideline, where it isn’t

clear what a standardized task sequence is.

These particular participants viewed the site as they talked, using a working definition of a task sequence, but never reaching certainty. Their assessment depended on their own understanding

Chapter 4 118 of a standardized task sequence, which may have differed from that intended by the author of the guideline.

Effect of designer experience on interpretation

Another area of disagreement between how participants interpreted a guideline emerged from a conflict between the expert opinion of the participant and the guideline itself. The participants were all experienced designers. Reading a set of guidelines and attempting to assess ten web sites did not mean that their experiences as designers could be easily put aside. As a result, guidelines which contradicted or questioned the experiences of the participants as designers were regarded with some suspicion. Guideline 11:8 (“Use at least a 12-point font (e.g., typeface) on all Web pages”) represents perhaps the clearest example of this issue. All four pairs of participants found the guideline problematic. As one pair of participants said,

(Participant A) It [the site being considered] does not meet the guideline, but I don’t think

that’s a good guideline.

(Participant B) I agree. I thought the page was totally readable, so the guideline sucks.

Another pair had similar reservations about the guideline. Following a discussion of how the guideline should be interpreted and applied, came these comments from one participant,

(Participant C) … I think that it depends on what the text is supposed to do. If the copy is

for reading, then yeah, sure. But if you’re typing a caption, or other text that is not very

Chapter 4 119

long or not very important, then using a smaller size is a good way to deemphasize the

text. It is difficult to create different levels of headings and that sort of stuff if you start

with a 12 point font, because 12 is fairly large.

The problem with the guideline which emerged in the discussion was that a 12 point font on one

monitor may appear as a smaller or larger font on a different monitor, depending on the screen

resolution setting of a particular computer. The designer has no control over the screen resolution

of the user of a site. As shown in the discussion of Guideline 8:1, agreement on the appropriate screen resolution (800 x 600 or 1024 x 768) was debatable. The font size on a page viewed on either of these resolutions would differ.

Web design poses unique challenges to designers when using terms and concepts developed in the medium of print. In print, type set in 12 point font does not potentially change as it might in electronic format, where facets of the medium (such as a computer screen resolution or browser settings) will affect the font size. In print, 12 point font is easy to measure and assess. This is not the case online.

Guidelines 7:8, 13:13, and 13:22 were also regarded with suspicion. One participant pair was clear in their issue with Guideline 7:8 (Keep Navigation-only Pages Short):

[Participant A] This is one I disagreed with. I remember thinking this is a bad guideline.

The guideline suggests that pages on which there is only navigation be kept relatively short, which might require using multiple pages. Participant B took issue with this suggestion,

Chapter 4 120

[Participant B] In some cases I think this [the guideline] is problematic. It is helpful to

have the navigation on one page no matter how long that page is, since you can use

search text to find it, whereas if it’s split on multiple pages it gets really annoying.

How their dislike of the guideline influenced their interpretation was unclear. While acknowledging their displeasure, they were still able to reach consensus. A dislike or mistrust of a guideline can confuse interpretation, as was the case with Guideline 13:13 (Use a Single Data

Entry Method). One pair was uncertain how to interpret a guideline which they believed

provided erroneous instructions,

[Participant G] This was not a good guideline. This one is pretty hard to follow. It

depends on the type of information. You could put form fields for dates, but drop down

helps to reduce errors.

[Participant H] And a change within a form shouldn’t be a huge problem. In fact, on this

site they have an example where they have… oh no, these are all drop downs. It does say

as long as possible, which keeps the door open.

[Participant G]: if you had to choose one [as a designer], you’d choose text fields for

everything.

Other participants also considered instances, outside of the sites under assessment, where

Guideline 13:13 suggested a problematic solution. The guidance to use a single data entry

Chapter 4 121

method contradicted the design experience of the participants. There were occasions in their

work where using different data entry methods improved usability.

[Participant E] The fact that you should use a single data entry method… means you have

to choose between items that can be in the same quiz. Some are meets and does not

meets, and some guidelines are wrong, or nonsense.

The frustration of this participant regarding guidelines was unmistakable. In a similar vein,

Guideline 13:22 (Use Data Entry Fields to Speed Performance) ran counter to the design

experiences of the participants. One participant was clear, “This was another guideline that I object to.” The participant went on to describe their professional experience regarding the use of data entry fields,

... if you were an experienced user, you could probably type dates into text fields

faster, than selecting them from drop down. But it’s way less error prone to use

drop downs and certainly there is a cognitive load to figure out what date format

to add into a text field.

The objections to the guideline were met with a similar reaction from the other participant.

Further, they struggled to find examples of where the guideline would be applicable and to agree on an interpretation for a guideline that ran counter to their own experience as designers. The struggle to find examples was evident in one pair’s discussion of Guideline 2:5 (Design For

Working Memory Limitations).

Chapter 4 122

[Participant G] This guideline I found peculiar. I was trying to think of examples of the

things that wouldn’t violate the guideline, and the only thing I could think of is when

you’re moving from screen to screen and you totally need to remember content from one

screen to another screen.

[Participant H] If you’re browsing through content, I could see where it would be used.

[Participant G] The information needs of the audience are important, regarding this

guideline. Do I need to memorize information from place to place on the site. Seems

impossible that you would never have a case where you didn’t use memory. I find the

guideline very vague. Do not require the user to remember information as they move

from place to place. What it suggests is impossible. When I browse the web I’m

constantly remembering what I see on one page. I see something, I Google it, I look

something up. That’s the nature of learning or reading.

Describing a guideline as “peculiar” or “vague” suggests an attitude more accepting than descriptors such as “nonsense” or “wrong”, but in terms of interpretation, such attitudes are just as problematic. Guideline 2:5 is yet another guideline which is unsupported by the experiences of the participant designers. The participants do not report a vague or abstract unease with

Guideline 2:5, but rather a clear sense that the guideline was problematic, “what it suggests is impossible.” The perception that the guideline recommendation is impossible comes from the participant’s experience as a designer. Where there is a mismatch between experience and

Chapter 4 123 guideline recommendation, the participants questioned the guideline.

Conflicts within a guideline set

Another factor influencing interpretation was conflict between guidelines in the guideline set. In some instances guidelines were perceived by participants as contradicting each other. Guideline

8:3 (use scrolling pages for reading comprehension) and Guideline 8:4 (use paging rather than scrolling) were discussed by all four participant pairs.

[Participant F] It does make for some confusion.

[Participant E] It shows how strange it is to put in the same guideline set two

contradictory guidelines, and ask for both to be present. It doesn’t make sense. It gives

the designer the means to be able to do both things and still get away with it.

Guidelines 8:3 and 8:4 include conditional statements; instances where either guideline should be considered by designers. Guideline 8:3 was intended for instances where users were required to read for comprehension and Guideline 8:4 depended on the anticipated system response time of the intended user. When one considers the intended use of the guidelines, they do not necessarily contradict each other, although not all participants interpreted the guidelines based on specific design conditions. One participant said, “I think these two guidelines contradict each other.

When do you not read for comprehension?” A participant in another pairing also noted confusion between Guidelines 8:3 and 8:4, and then criticized Guideline 8:4,

Chapter 4 124

[Participant G] Use longer scrolling pages when users are reading for comprehension.

Which I find funny that 8:4 is “use paging instead of scrolling.” It’s not clear. It says for

response... [reading] ... for quick system response. But if you’re talking about text copy –

you’d have to have a very long page before response becomes an issue.

Criticism of Guideline 8:4 was evident in the comments of another participant pair as well, who

based their interpretation on the specific content on About.com/Health, where articles were

relatively short. The length of the articles became a determining factor in the discussion,

[Participant A] The articles are quite short, that’s why I put MEET. If you broke it apart, it

would be harder to read.

The apparent conflict was sometimes more subtle. One participant found conflicting advice

between Guideline 6:9 (“Avoid Scroll Stoppers”) and Guideline 6:11 (“Use Moderate White

Space”). A “scroll stopper” is a visual element on a page that potentially misleads the user into believing they have reached the bottom of the page when that is not the case. The participant said, “there is a connection between 6:9 and 6:11 in how the white space might have made other

elements on the page not seem like scroll stoppers.” The degree of white space used on a page

might mitigate the problems with elements that appeared to be scroll stoppers. For this

participant, following the advice of the latter guideline might cause problems with the former

guideline. Referring to the examples provided in Guideline 6:9 did not help with the

interpretation.

Chapter 4 125

Use of examples within guidelines

Examples are intended to improve clarity. However, guidelines with high rates of agreement and those with low rates (see section 4.3.3 above) could not be differentiated in terms of the number of examples provided, which questions the usefulness of examples. All participant pairs made reference to the use of examples in clarifying the intention of a guideline.

Guideline 6:9 addressed the issue of scroll stoppers; design elements that mislead the user of a site into thinking they have reached the end of the page. One pair of participants agreed with the guideline’s instructions to “Ensure that the location of headings and other page elements does not create the illusion that users have reached the top or bottom of a page when they have not,” but they did not agree on how to interpret this guideline for specific sites.

[Participant B] I felt that I still understood I should go down the page. They weren’t

scroll stoppers.

[Participant A] (reads guideline). For me, the sponsor link, because of the lines,

constituted a scroll stopper.

[Participant B] I don’t agree. The way the content is staggered on the page, you can still

see that you can go further.

[Participant A] There is no guideline about staggered content. Because of the lines….

Chapter 4 126

In one case, when discussing www.cancer.gov, the example provided by a guideline was considered too subtle and not clear enough to be of help. Guideline 11:10 (“Emphasize

Importance”) provides an illustrated example drawn from the Department of Defense (U.S.A.) website.

[Participant B] What would you emphasize on a site like this? The example in the book

doesn’t really help. Based on a health site, this is attention getting.

[Participant A] It’s hard to tell. Considering the example, then this would be meet. If the

example is the bar to jump to, then this would meet. If it were a title in a white box, it

wouldn’t meet. The example is subtle and it does meet.

The illustrated example in Guideline 11:10 shows the title in a grey box, below which are two columns of underlined hypertext links. There are some bold links, to which the guideline explicitly refers as “effectively emphasizing important topic categories” (U.S. Department of

Health and Human Services, 2006. p. 109). Guideline 8:1 (“Eliminate Horizontal Scrolling”) used two screen captures to illustrate the guideline, but the resolution used led one participant to remark, “I disagree with this guideline, or rather the example they had. A 640 by 480 screenshot, but I based my assessment on 800 by 600.” As discussed earlier, Guideline 2:3 (“Standardizing

Task Sequences “) was unclear to one participant pair. The use of examples (in this case two different illustrated examples of interactive calendars from travel websites) did not help clarify the interpretation. “I think this guideline is too specific to a travel site,” said one participant. “I could see it making sense there. If you take this outside of the example, then nothing on the site

Chapter 4 127

is standardized.” The illustration was context specific and considered of limited value by

participants as they assessed cancer health information sites. Another participant remarked that,

“The problem with this is that the example implies you should be looking at calendars.” Again, a

different context lessened the value of the illustrated example.

Examples also run the risk of confusing the reader. When discussing whether

www.cancer.ca met Guideline 16:9 (“Avoid Scroll Stoppers”), one participant remarked,

I’m looking at the guideline and think I interpreted it as being not related to text, because

I think there is another guideline section that does that. Because the example is of a map,

I was thinking more of graphics. But on the guideline itself, it doesn’t really say

specifically referring to graphics. The guideline sounds fairly general.

The participant seems to understand that the guideline does not refer to graphics, but because the example is of a graphic (a map), the interpretation is called into question. Perhaps a better example could have been chosen. Interestingly, there are no references in the discussion to examples which helped clarify a guideline, perhaps because a clear illustration did not lead to disagreement amongst the participants.

The discussions of disagreements within the pairs of participants revealed five key issues which have been the focus of section 4.3.4. The issues regarding interpretation of guidelines overlap and are interconnected. A user of guidelines must first understand what constitutes adherence or non-adherence. This understanding can hinge on a word or key phrase.

Interpretation of the guideline is also affected by the previous experiences of a designer, particularly if the guideline recommendation runs counter to design practices. The interplay of

Chapter 4 128 guidelines also influences interpretation, particularly if the user of a guideline set finds contradicting advice. The study found instances where the examples further complicated interpretation, rather than helping to clarify the intention of the guideline.

While focusing on problems with the interpretation of guidelines highlights specific issues, the study found that the guideline set as a whole was well received by the participants.

They appreciated the opportunity to explore a guideline set which was unknown to any of them prior to taking part in the study. As one participant said, “The guidelines are good. But the problem with the guidelines comes in taking them too literally. Grammar guidelines are similar, in that any great writer will break those rules.”

The participants understood the importance of interpretation, as well as understanding that guidelines are not rules. The interpretation of guidelines requires some flexibility. As one participant stated regarding the application of guidelines, “A lot of it is up to interpretation. If you try to define it too much, you might get something too rigid.”

4.4 Summary

As the study was first being designed, it was believed that although participants would have conflicting assessments, in pairs they would resolve these conflicts and that all four pairs of participants would report almost identical assessments. An earlier study which also paired two evaluators after they conducted individual evaluations found the level of disagreement range as high as 51%, but the conflicts were quickly resolved during discussion between the two evaluators (Cherry, Muter & Szigeti, 2006). Following the findings of this earlier study, it was assumed that different pairs would reach similar interpretations of the guidelines.

Chapter 4 129

During the meetings of pairs, participants in the current study combined to change almost one third of the total assessments (as shown in Table 4.5, the lowest number of disagreements between a pair of participants was 216 (Pair GH), whereas the highest was 234 (Pair AB) from a total of 670 assessments). Yet these changes still resulted in a compliance rate very similar to that found in the individual assessments. Individual participants reported between 50.0 % and

73.9% compliance. In pairs, the participants reported between 52.2% and 73.7% compliance even though changes were made to between 216 and 234 guideline assessments. This suggests that the end result in terms of the percentage of compliance is not changed by consultation or partnership of evaluators. While assessments based on specific guidelines changed, considered in aggregate the site rankings altered little through the process of participant discussion. The data shows that if there are specific issues with a website, discussion between two evaluators does not necessarily improve the compliance rate. Overall guideline adherence per site did not change following paired discussions of the individual assessments. The interpretation of specific guidelines, on the other hand, did change following discussion.

The study explored differences across the guidelines in terms of their characteristics. The task of synthesizing multiple research studies and reducing the findings to a guideline is certainly a daunting task. The process of creating a guideline places as much importance on what information is included as it does on what information is left out or implied. It was hypothesized that guidelines on which there was greater agreement during the assessment of websites would differ from guidelines with lower agreement rates in terms of their characteristics, but this was not the case. A comparison of six guidelines with high rates of agreement to eight guidelines with low rates of agreement found no notable differences. If there was a reason that some guidelines found higher rates of agreement amongst participants, it was not found in the

Chapter 4 130

characteristics of the guidelines themselves. Comparison revealed that the characteristics of

guidelines on which there was agreement in interpretation were similar to guidelines on which there was very little agreement. Rather, it was the analysis of participant discussions which revealed subtle differences between guidelines in terms of how they were interpreted and understood by participants.

de Sousa and Bevan (1990) and Tetzlaff and Schwartz (1991) found that designers only

partially understood guidelines and found some of them problematic (if not incorrect). These

findings were supported by this study, both in the inconsistency of interpretation and the paired

discussions where participants argued against the validity of some of the guidelines. Phrases such

as “I remember thinking this is a bad guideline,” or “some guidelines are wrong, or nonsense”

speak to issues that Tetzlaff and Schwartz described over two decades earlier.

When participants felt a guideline recommendation ran counter to their design

experience, they expressed displeasure and mistrust. The ideas presented within guidelines,

regardless of their provenance and strength of evidence, had to agree with the design experiences

of participants. The participants were all professional designers. A guideline, to be accepted

without mistrust, had to provide solutions similar to what the participants’ experiences had

already confirmed. A disconnect between advice provided by a guideline and the experience of a

designer led to disagreement and potential problems with interpretation. Participants in the study

struggled when a guideline appeared to be incorrect. When a participant suggested that Guideline

13:13 is, “not a good guideline ... It depends on the type of information. You could put form

fields for dates, but drop down helps to reduce errors,” they are referencing their own

experiences with the design problems addressed by the guideline. Form fields can work, but their

experience tells them that drop down menus help resolve another problem; the introduction of

Chapter 4 131 erroneous data. The discussion of Guideline 2:5 served as an example of how designers struggled with a guideline that offered advice that appeared to contradict their own experiences. They did not reject the guideline, but rather called it “peculiar” and worked to find an interpretation within the context of their own experiences. When a participant stated that they find “... the guideline very vague ... What it suggests is impossible,” they are struggling to interpret the guideline in a manner that conforms to solutions they have employed in their own work.

The mistrust was further heightened by guidelines which were perceived by participants as conflicting not with design experience, but with other guidelines within the set. If guidelines themselves are not in agreement, why would the designer reference a guideline over their own experiences? Such conflict greatly lessens the value of the guideline to the designer. As one participant stated, conflicting guidelines, “give the designer the means to be able to do both things and still get away with it.” It is important to note that guideline conflict was a matter of interpretation; viewed differently, the guidelines may have complemented each other. But if the interpretation of the guideline is such that the design solutions suggested are in conflict, then there is a danger that the guideline set as a whole may be mistrusted. As Tetzlaff and Schwartz

(1991) found, the guidelines may have been misinterpreted by the participants, which is a problem which certainly might occur outside of a research study. How to clarify misinterpretations remains unclear, particularly where a flawed example or a single word influences how a user might perceive the intentions of a guideline.

The study found examples where the interpretation of guidelines by participants depended on their interpretation of a single word or phrase. The presence of ambiguous words, such as “appropriate” or “important” required a greater effort of interpretation than terms such as

“use” or “eliminate.” When asked to assess a site against a guideline, ambiguity can lead to

Chapter 4 132 differences. Even terms that may arguably be considered technical, such as “category” or

“standardized task sequence” presented problems to practitioners who understood jargon. If the translation of research findings to guidelines necessitates the use of terms which are relative to different situations or applications, then the result will be differing interpretation. What is

“appropriate” varies from one guideline user to another. It may be argued that such terms should be supported by examples, but the study found that examples did not lead to clarity.

Examples, particularly illustrated and annotated examples, should help clarify a guideline. But the study found that sometimes they have the opposite effect. When an example for a guideline that considers text uses graphics for illustration, the guideline itself may be compromised. One participant’s interpretation differed from the participant with whom they were paired solely because of the example employed in the guideline. The participant said that,

“I’m looking at the guideline and think I interpreted it as being not related to text, ... because the example is of a map, I was thinking more of graphics.” This interpretation led them to make a different assessment from the participant with whom they were paired, an assessment that was changed following discussion about the illustrated example and its relevance to the guideline.

The guidelines in the Research-based Web Design & Usability Guidelines set (U.S. Department of Health and Human Services, 2006), with some exceptions, included one or two examples, so it is not possible to compare guidelines with one or two examples to those that had five or more.

Would more examples improve interpretation? If so, how many examples would be ideal?

The study found, depending on the specific guideline, a narrow definition of conditions was favoured by the participants. Guideline 11:8 called for the use of minimum 12 point font. If participants found an example of smaller fonts, they did not consider the site in compliance with the guideline. The approach of the participants at first seems clear; a state of either/or. But two of the pairs complicated the interpretation by introducing an element not discussed in the guideline;

Chapter 4 133 the presence on the website of a widget which allow the user to alter the size of the font. This was a method for addressing issues of readability. Ensuring readability is the goal of Guideline

11:8 and the presence of such a widget made it possible to meet this goal. Yet this interpretation is problematic for two reasons; (i) the ability to change font size is not part of the guideline and

(ii) research studies may find that users do not favour the use of such applications, in which case the ability for users to change the size of the font may not be a solution to the problem raised in

Guideline 11:8. This illustrates that the ways in which guidelines may be interpreted is complex and unpredictable.

Initially, the goal of the study was to produce an assessment of all ten cancer health information sites against a guideline set without any disagreement. It was hoped that participants would reach consensus not only in pairs, but across pairs. While some variation in interpretation was expected, the results following the paired participant discussions were unexpected; there remained a disagreement rate of almost one third among the four participant pairs. If it were possible, as originally intended, for the participants to be brought together to discuss these disagreements, would consensus have been reached for all 670 assessments? There is no reason to believe that a three-step process (individual assessments followed by paired assessments followed by a group assessment) would result in a more valid assessment of the sites. Even recognizing that designers sometimes work in pairs or groups, it is doubtful that such a three-step process represents how guidelines are used by the design community.

It is unclear what affect the context of health care information had on the research. The

U.S. Department of Health and Human Services (2006) guideline set was based on the National

Cancer Institute (2001) guideline set. The use of a guideline set, which emerged from an institutional body focused on cancer research, to assess health information sites focusing on cancer, may be an appropriate pairing of guideline set and design context, but it is important to

Chapter 4 134 recognize that this is only one information context in which designers might work. Whether a different guideline set or a different set of websites would result in different findings is unclear.

While the discussions of eight designers allowed for an understanding of the role interpretation plays in understanding guidelines, how designers use guidelines, which guidelines designers are aware of and what designers think of guidelines remains unknown. The next chapter presents the results of a study involving the broader design community, where 116 participants shed more light on the awareness and use of guidelines by web interface designers.

Chapter 4 135

Chapter 5

Designer awareness, perception and use of web interface design guidelines (Study #3)

5.1 Introduction

The first study, described in Chapter 3, considered the attitudes of design students towards guidelines and revealed that the favourable use of guidelines was predicated, in part, on the designers’ existing experiences. The study also identified the need for further research, in particular a broad study of the design community regarding guidelines. A web-based survey was designed in order to better understand the perceptions and attitudes of professional web interface designers.

By exploring the attitudes of designers towards the application of guidelines to their design work, the study addressed five of the research questions proposed in Chapter 2:

RQ1: What are web interface designers’ attitudes towards the use of design guidelines?

RQ2: What is the level of awareness and usage of evidence-based web design guidelines by

web interface designers?

RQ4: When are guidelines used in the design process to solve design problems?

RQ5: What specific design trade-offs influence the work of designers?

RQ6: How is knowledge and information shared amongst web interface designers?

Chapter 5 136

5.2 Methodology

A web-based questionnaire was developed, tested, and posted to collect data over a three week

period. Web interface designers were invited to participate in the study via email, notices on

listserv and message boards, as well as through snowball sampling. As the total number of

individuals who practice web design is unknown, there is no sample frame.

5.2.1 Method

The aim of the web-based questionnaire (see Appendix G) was to collect the opinions of web

interface designers regarding interface design guidelines and design trade-offs. A web-based

survey was chosen as the method for reaching the study population for a number of reasons; the

geographic dispersion of potential respondents, the measure of anonymity afforded by the

instrument, the reasonable assumption of both technical expertise (familiarity with the internet)

and internet access on the part of web interface designers, and the ability to reach a large number of respondents. The lack of a certifying body to either define requirements of membership or provide a sample frame created challenges (contrasting examples include the Professional

Engineers Ontario, Ontario Bar Association and Ontario Medical Association which perform this function for those respectively practicing engineering, law or medicine in the province of

Ontario). Associations which do exist have loosely defined criteria for membership and rely on self reported membership qualification (The Interaction Design Association, 2011; InfoDesign,

2011).

Web interface designers, regardless of association or membership, work online, and a web-based survey allows participation via a method of communication central to their

profession. A survey also allows for anonymity, which affords the opportunity to ask

Chapter 5 137

respondents questions, such as those involving personal knowledge or opinion, which may be

negatively biased by the presence of a human interviewer (Dillman, Smith & Christian, 2009).

Like many occupations, web interface designers operate worldwide. The location of web interface design work is limited only by internet access, and even a study focused on English language respondents can reasonably expect international participation. Web-based surveys have

been used by other researchers attempting to reach similar potential respondents, including web

developers in academic libraries (Connell, 2008), and members of the broader engineering

design community (Karam, O’Connell & Perry, 2009).

5.2.2. Apparatus

The limited research in this area meant that there were no existing tools which could either be

used in their entirety or adapted for this study. However, where possible, demographic questions

were drawn from existing web-based surveys (Karam, O’Connell & Perry, 2009; Connell, 2008).

The beta version of the questionnaire comprised of 32 questions and was posted online using a

proprietary third party service (www.surveymonkey.com). This service was chosen because of

its ease of use, ability to customize the interface, and familiarity to potential respondents.

The questionnaire was divided into four sections: general questions regarding the nature

of the respondents’ design work, questions pertaining to the use, familiarity, and attitudes

towards web design guidelines, questions which considered web design trade-offs, and

respondent demographic information. The questionnaire also included a request for additional participation in the form of a follow-up phone interview. The beta version of the survey instrument was tested in order to improve questions, scales and the overall format. The instrument was tested by individuals with experience in developing web-based questionnaires,

Chapter 5 138

with web interface designers (who represented potential respondents) and a combination of both.

Testing was done online both remotely and locally. The remote testing further used two

approaches; one with an evaluation tool for guidance (see Appendix H) and one without any

additional guidance, in order to ascertain how long it would take to complete the questionnaire.

Local evaluation used a think aloud method, where evaluators verbalized their thoughts while

completing the questionnaire with the researcher present. Thirteen individuals participated in

testing the instrument (comprised of four researchers with experience developing survey

instruments and nine web interface designers). Testing was done in sequential order (as opposed

to multiple tests conducted in parallel), which allowed for improvements to the questionnaire following each test. The iterative testing resulted in numerous changes, including the removal of

two questions, modification of open to closed questions, and revisions to the wording of both

instructions and specific questions.

5.2.3 Respondents

Respondents were recruited from the web interface design community via email (from listserv

and membership lists) and discussion board postings. An invitation to participate was posted on

three occasions on the Association of Computer Machinery SIGCHI membership listserv,

specifically targeting two mailing lists with a combined membership of just less than 5000: ACM

SIGCHI General Interest Announcements and ACM SIGCHI WWW Human Factors. An

invitation was also posted on the IxDA (Interaction Design Association) discussion board, where

both a follow up message and a reminder were posted during the survey period. IxDA has an

estimated membership of over 10,000 individuals. Two members of the design community used

social networking tools to raise awareness of the study; Kelly Lyons (via Facebook) and Karel

Chapter 5 139

Vredenburg (via Twitter and his blog). In addition, colleagues from the design community were

asked to share contact information (a name and an email address) of any web interface designers

they knew. This method of snowball sampling resulted in a list of 131 individuals who were then

contacted via personalized email. In all cases, the invitation to participate specified that

respondents should be web interface designers and that the study was “interested in the

experiences you and other web interface designers have with aspects of the design process, such

as the use of guidelines and your experiences (if any) with design trade-offs.” Individuals were

asked to volunteer and complete a web-based questionnaire approximately 20 minutes in length.

Respondents were not compensated for participating.

5.2.4 Procedure

The web-based survey was active for a period of three weeks. Participant invitations included a

URL to the survey tool. The first question, “Do you currently design web interfaces?”, was used to disqualify those individuals who did not answer affirmatively. Information on consent was provided on the first screen of the questionnaire, and a question was also included at the conclusion of the questionnaire asking for respondent consent to use the data. Once complete, the respondent was directed to the researcher’s website (www.szigeti.ca) where information on guidelines and similar tools was made available. An executive summary of the survey results was posted on the researcher’s website two months following the close of the survey (September,

2010).1

1 Both Study #3 and #4 (see Chapter 6) were approved by the Office of Research Ethics at the University of Toronto.

Chapter 5 140

5.2.5 Measures

The survey instrument asked 37 questions. There were 21 closed questions and 13 open questions. In addition, two questions asked respondents if they would take part in a further study

(see Chapter 6), and one question asked for permission to use the study data.

5.3 Results

The data collected in the survey will be discussed in four sections. The first section focuses on the

respondents, including their employment, education and experience as practitioners. The second

section focuses on respondents’ awareness and use of guidelines. Attitudes towards web design

guidelines are considered in the third section, and the final category considers design trade-offs.

5.3.1 Demographic Data

The study captured data from 116 respondents, of whom 59 self identified as Canadian residents,

40 as residing in the U.S., five in India, and the remainder in the U.K. (four respondents),

Australia (two respondents), Germany (two respondents), and a single respondent in each of

Austria, Croatia-Herzegovina, and France. One participant chose not to indentify a country of

residence. Sixty-seven of the respondents were male, 42 were female (seven respondents chose

not to report their gender). Over two thirds of the respondents (69.9%) were between 25 and 44

years of age. Table 5.1 shows the age range of respondents, including three individuals who

preferred not to provide their age range, and four who skipped the question.

Chapter 5 141

Table 5.1: Age range of respondents Respondent age range Count % of total Under 25 5 4.3% 25‐34 43 37.1% 35‐44 38 32.8% 45‐55 15 12.9% Over 55 8 6.9% Preferred not to answer 3 2.6% Skipped question 4 3.4% Total 116 100.0%

The experience of respondents was measured in three different ways; (i) by the number of years working as a web interface designer, (ii) by the involvement, either in a lead role or as a participant in projects of various budgets, and (iii) by asking respondents to describe their experience. The web interface design experience of respondents varied from less than one year to over 25 years. Almost a third of the respondents reported between 11 and 15 years of experience.

Web interface design is a relatively new area of employment, which began to emerge after the introduction of browsers which supported graphic files in 1994. Prior to the introduction of these browsers, internet use held limited appeal to individuals outside of military and academic interests, which possibly limits the number of web interface designers reporting more than 16 years of experience. However, as illustrated in Table 5.2, almost 7% of the respondents did indicate that their work started prior to the launch of these early browsers.

Chapter 5 142

Table 5.2 Experience of respondents as web interface designers

Experience as web interface designer Count Percentage of total less than 1 year 5 4.3% 1‐3 years 16 13.8% 4‐6 years 17 14.7% 7‐10 years 32 27.6% 11‐15 years 38 32.8% 16‐20 years 7 6.0% 21‐25 years 0 0.0% more than 25 years 1 0.9% TOTAL 116 100.1% due to rounding

Another measure of experience is the size of the projects in which respondents were involved.

Small projects (less than $20,000) differ from large projects (over $500,000) in a variety of

ways, potentially including the size of the design team, timelines, client expectations and resources. Figure 5.1 illustrates respondent involvement in projects of varying budget levels.

Chapter 5 143

100% 90% 80% 70% 60% participants 50% of

40% Lead Role 30% Participant 20%

Percentage 10% 83.5% 57.7% 74.4% 58.5% 59.2% 68.4% 56.3% 75.0% 58.2% 72.7% 0% Less than $20,000 to $75,001 to $200,001 to Over $500,000 $20,000 $75,000 $200,000 $500,000 Size of project budget

Figure 5.1: Involvement of respondents in projects of varying budgets

Over half of the respondents played a lead role in projects with budgets over $500,000. The years of experience reported in Table 5.2 combined with the variety of project involvement reported in the table above suggests diversity in the aggregate work experience of the respondent group.

The breadth of experience was evident in the descriptions provided by respondents. Some wrote of their work in minimal terms, “mostly basic website layout and navigation setup,” or

“simple web design,” while other respondents described what might be similar work in more detail, for example,

Determine overall functionality, identify what the site needs to accomplish,

identify private versus public faces, determine look and feel elements, prepare

general layout to visualize look and feel and graphic elements, identify resources

required, prepare budget estimates, act as project manager.

Chapter 5 144

Respondents referenced specific domains of design work, including e-learning, e-commerce and

health care and involvement in very specific design tasks, including , building

browser based tools, connecting front and back end applications, and working on a number of

devices utilizing web interfaces such as desktop computers, iPads, laptops, and mobile devices.

The following respondent described what they felt were common tasks; tasks which indeed were

echoed in the answers of other respondents,

The following tasks are common in my work, yet sometimes are absent of a

project due to its budget/complexity required/primary point of contact): Receive

briefing from project managers or directly from clients, quote project (RFP),

establish art direction by supplying mood board and preliminary of a

very generic interface of the site (in Photoshop), create architecture and

wireframes (in Omnigraffle or Illustrator), create main graphical mockups in

Photoshop of the homepage and one landing page (this includes multiple versions

on the graphical theme and often the art direction is established at that time), wait

for approval, modify maquettes until final approval, apply graphic scheme to

other key pages (pages that are unique in themselves and need to be mocked up

for coding purposes), send PSD [Adobe Photoshop] files to integrator/coder.

The list of respondent experiences related to web interfaces covered a large number of areas where design problems were identified, including user flow diagrams, wireframes, data analysis, functional prototyping across narrowly defined environments such as AJAX and mobile Apple

Chapter 5 145

products. While some respondents focused on specific aspects of their work, one respondent

simply described their web interface design work in broad terms, “I redesign poor interfaces.”

Respondents were asked about their employer in terms of company size (as measured by the number of employees) and the number of web designers employed by the company. Figure 5.2 shows that 49% of respondents worked at companies with at least 51 employees, while 21% were self-employed.

Self employed over 1000 21% 28%

2 to 10 501 to 1000 19% 9%

101 to 500 11 to 50 8% 11% 51 to 100 4%

Figure 5.2: Number of employees where respondent works (n=116)

Chapter 5 146

The number of web designers employed by a company may influence knowledge sharing within an organization. Figure 5.2 shows that nine percent of respondents were the only web designers

in the company, whereas 45% of respondents indicated the employment in their company of at

least five web designers in addition to themselves. Twenty-six respondents skipped this question.

None 9% 15 or more 23% 1 11%

10‐14 8% 2‐4 5‐9 35% 14%

Figure 5.3: Number of web designers employed in addition to respondent (n=90)

An emerging discipline such as web interface design encompasses practitioners with varied educational backgrounds. Figure 5.4 illustrates nineteen areas of specialization and five levels of accreditation from self-reported data.

Chapter 5 147

Number of respondents 0 5 10 15 20 25

Anthropology Certificate Applied Arts Diploma Architecture Business Administration Undergrad Cinema & Photography Masters Communications Computer Science PhD Design Education Categories

Engineering Literature Fine Arts Human Computer Interaction

Education Information Studies Landscape Architecture Mathematics & Physics Psychology Self‐Taught (Autodidact) Undefined

Figure 5.4: Education of respondents

Respondents listed their certifications, diplomas or degrees (if any) and the areas of specialization. The various ways in which similar degrees were identified meant that the categories in Figure 5.4 conflated reported data. For example, the category Information Studies encompasses terms such as Library Science and Information Technology. Categories such as

Computer Science, Design, and HCI combined represent only one third of the total reported accreditations. Thirty-seven percent of the respondents indicated that accreditation was at the graduate level (masters and Ph.D).

The respondents were not a homogenous group. As Tables 5.1 to 5.2 and Figures 5.1 to

5.4 illustrate, they can be differentiated by education, years of experience, geographic location, and company size. Before considering their attitudes towards guidelines, their awareness and use of guideline sets will be discussed.

Chapter 5 148

5.3.2 Guideline awareness and use

Eighty-three percent of respondents indicated that they had used web design guidelines in their

work. When asked to list specific guideline sets, 12% of respondents reported the use of in-house

guidelines. While some respondents described individual guidelines addressing narrowly defined

design problems, a number listed broader guideline sets; 23% referenced W3C guidelines, 14%

referenced Nielsen’s Heuristics and 6% referenced the Research-based Web Design & Usability

Guidelines (U.S. Department of Health and Human Services, 2006). These guidelines were listed prior to the question which asked about specific guideline sets. Clients provided guidelines for design projects in 11% of the cases and 7% of respondents reported pattern libraries as the source of guidelines used in their work. As one respondent noted, “A lot of terms used in this field do not have universal meanings to everyone.”

Respondents were asked about their level of awareness regarding five specific guideline sets; ISO9241 Ergonomic requirements for office work with visual display terminals (VDTs),

Research-based Web Design & Usability Guidelines (U.S. Department of Health and Human

Services, 2006), Research-based Web Design & Usability Guidelines (National Cancer Institute),

Web Content Accessibility Guidelines, version 1.0 (W3C) and Web Content Accessibility

Guidelines, version 2.0 (W3C). Figure 5.5 shows that while awareness of the ISO9241 guideline sets are the highest among the five listed, the least number of respondents use it. The set used by the largest percentage of respondents is the Web Content Accessibility Guidelines, version 1.0 with 47% of respondents reported having applied these guidelines.

Chapter 5 149

90%

80%

I have applied guidelines 70% from this set

ondents 60% p I have read this set

res 50%

of

e 40% g 30% I am aware of this set

20% Percenta 10%

0% WCAG, WCAG, Research‐based ISO9241 Research‐based v.2 v.1 Web Design & Web Design & Usability Usability Guidelines Guidelines (HHS) (NCI)

Figure 5.5: Awareness and use of specific guideline sets

Respondents learned about guideline sets through a variety of sources. Figure 5.6 lists 14 sources, as well as a fifteenth category where respondents could specify other sources. Almost

60% of respondents learned about guideline sets through reading blogs, 56% credited other designers and 55% learned about guideline sets via books.

Chapter 5 150

Other Designer Blog Book Search engine Employer Educational Institution Client Listserv or message board Journal (online) Other e‐newsletter Journal (print) Podcast Consumer Magazine Newspaper

0% 10% 20% 30% 40% 50% 60% 70%

Percentage of respondents

Figure 5.6: How respondents learned about web design guideline sets

Twenty-one percent of respondents selected the category “Other.” These responses included conferences, websites, self study, and social networking tools (such as a Wiki site and Twitter).

When asked if there were guidelines that respondents were aware of and did not use, 57% reported positively. The reasons for not using these guidelines varied. Respondents who wrote,

“W3C is sufficient,” or “no need” spoke to the focus of their work domain. The confines of the work domain were echoed in statements such as, “some W3C standards, for example, avoiding the use of Flash, are too stringent for our needs” and in the words of a respondent who listed specific guidelines that were not used, “Section 508 and EU directive on accessibility... I don't do a lot of US & EU government work.” Other respondents were aware that the guidelines they

Chapter 5 151 used either incorporated parts of existing guideline sets they could identify, “there are plenty of published guidelines – Nielsen, Spool, etc. – and standards that work from principles we have included in internal guidelines,” or felt there were too many to list. “Too many to identify really,” wrote one respondent, “I use components of any I see that I like.” Another respondent cited a lack of internal support for guidelines (“There are guidelines for accessibility I'm aware of, but haven't used at work because I can't convince others to support the work.”) whereas another designer shared a skepticism regarding guidelines,

I generally resist guidelines when they're a requirement given to me by clients.

Govt institutions are especially bad for creating systematized, rigid guidelines,

and a dogmatic insistence on their implementation. I often go along in the end but

I avoid them because they don't allow for innovation and creativity. They are also

often based on a lot of unproven assumptions.

Skepticism about guidelines was evident in the words of various respondents. A number of responses in the survey suggested that guidelines had to be used with caution, often because they ran counter to the experience of the designer. One respondent offered advice regarding the use of guidelines,

To my knowledge there are many usability guidelines that in my opinion, if

followed strictly, do not always contribute to a fulfilling user experience or meet

strategic objectives. Based on the project objectives and user needs I seek to find

the best solution for my user audience based on their emotional, social and

Chapter 5 152

practical needs. Having said that, I feel it is important for every web designer to

have a good understanding of basic usability heuristics and always design with the

user in mind. Keeping up with design trends and technology is also of vital

importance.

The next section reports on respondents’ attitudes towards guidelines and their value as a .

5.3.3 Attitudes towards web design guidelines

Whether guidelines are perceived as helpful in solving design problems is unclear in the literature. Tetzlaff and Schwartz (1991), as well as Kim (2010), reported that designers were mistrustful of guidelines, particularly if the guideline suggested actions that were contrary to their experiences. In order to better understand the attitudes of designers towards guidelines, the study asked respondents if they felt guidelines were helpful, and to describe how they were helpful, or in cases where they were not, why this was the case.

When asked how frequently guidelines have helped address design problems, over 41% of respondents indicated “sometimes”, compared to almost a third who selected “often”, as shown in Table 5.3. No respondents selected “Never.”

Chapter 5 153

Table 5.3 Frequency with which guidelines help address design problems

Response # of respondents % of respondents Often 32 27.6% Sometimes 48 41.4% Rarely 12 10.3% Never 0 0.0% Don’t Know 4 3.4% Skipped question 20 17.2% TOTAL 116 99.9%

The frequency of guideline use was not dependent on the number of employees at an

organization or the age of the respondent (SPSS software was used to cross tabulate guideline

use and organization size.)2 In describing how guidelines were helpful, responses addressed one

of four categories; (i) by imposing standardization on designs, (ii) by providing a framework for design projects, (iii) through supporting design choices and (iv) by functioning as a checklist.

Even if guidelines are not perceived as standards or rules, the designer can use them to impose design limitations. If the solutions to problems seemed overwhelming, guidelines were helpful in, “reducing choice by applying a standard approach.” The idea of standardizing the design and reducing choice through simplification was viewed as helpful. “A standard approach ensures consistent design,” wrote one respondent, who felt a guideline “simplifies” design problems with the caveat that guidelines, “need to be flexible for when the standard does not fit.” The notion of flexibility differentiates a guideline from a rule or law. One respondent wrote that,

2 Not surprisingly, employer encouragement of guideline use positively influenced the frequency of use. Cross tabulation of employer attitudes towards guideline use and the frequency of guideline use found that respondents who were not self employed were predisposed to use guidelines more frequently if they perceived their employer to encourage the use of web design guidelines, then those who did not perceive such encouragement.

Chapter 5 154

The guideline indicates in many cases how to address a problem but not the actual

solution. As long as the core structure of the guideline is kept intact, we

implement a solution that adheres to the guideline. In other cases we reference the

guideline to ensure we are not breaking standard UI paradigms.

Design can be perceived as the act of navigating through a series of potential problems, where

guidelines, standards and paradigms all play a role in finding resolution. One respondent wrote

that guidelines “provide standards, examples, and anti-patterns directly or indirectly related to

design challenges.” Another respondent wrote,

It helps to have a framework/restrictions when it comes to designing. Standard

answer, but happily a truth. Using web and accessibility standards helps to

simplify (sometimes) the approach.

Simplifying design choices is an important aspect of guideline use. Three respondents used the

term “framework” to describe a means to organize or structure the design choices and possible solutions offered by guidelines. One respondent wrote that guidelines, “give a basic framework on which to hang a design, the theory behind what works, etc.” A framework can also be used to support design solutions when interacting with both internal and external clients. In these cases, guidelines are not just helpful in providing design solutions, but also in supporting decisions.

Guidelines, “provide a framework for the basis of the design, but you usually have to be creative to get the final result. [Guidelines are] useful for justifying design decisions to management.”

Their helpfulness in this regard was obvious to another respondent, who felt guidelines were an aid,

Chapter 5 155

... to convince client[s]. Guidelines helped us to understand strength[s] and

weaknesses of alternative proposals I made. Guidelines helped us rapidly decide

between two or more design choices.

The use of guidelines to support design decisions or to support the designer’s expert opinion, was

referenced by other respondents. The designer might use their expert opinion in finding

solutions, but guidelines can strengthen the case for the solution in the eyes of the client, as

evidenced in the comments of a respondent, “The research-based guidelines provided an external

authority on a few occasions when I needed to backup my opinion. I recall emailing a client clippings of a few guidelines.”

Respondents suggested considering guidelines prior to starting a project, some felt

guidelines were good to use during the entire process of design, while others felt they were most

useful once the design was almost completed, as a means to check if the design had missed any

important considerations. Such use implies guidelines functioning as memory aids. Finally,

respondents also suggested utilizing guidelines only in support of designs when presented to

clients or peers.

Using guidelines in the latter stages of the design process, as a way to validate solutions

or as a memory aid, implies that they best be used as a checklist. The use of guidelines as a

checklist was suggested by respondents in Study 1 (see Chapter 3), where the issue of validation

and support was also raised. Checklists can support design choices (“They are most useful as a

checklist to vet my designs against”) and help designers compensate for the limits of memory,

Chapter 5 156

The most useful part of a design guideline is as a reminder of something I may

have forgotten. The principles are in my head from education and experience, but

sometimes I forget to consider something, so a quick check at a guideline or

checklist will refresh my memory.

However, not all respondents considered guidelines helpful. Twelve respondents raised issues

around the use of guidelines. Two reasons were indicated for finding guidelines “rarely” helpful,

(i) that guidelines are redundant, and (ii) that guidelines are too narrow in focus. Respondents

who perceived redundancy were not critical of the guidelines themselves, but rather either (i) had

other tools at their disposal,

It's not that guidelines didn't help, it's that prototyping, , feedback

from demos, and pilot studies were more helpful than any abstract, prescribed

guidelines.

or (ii) felt the design decisions can be addressed by expertise (“many design decisions are simply common sense”). This last opinion was echoed in the words of another respondent, who wrote that guidelines,

... often solve general problems that have obvious solutions. Whereas, I typically

have a specific problem with a-not-so-obvious solution. Design guidelines could

be more helpful if they provided a plethora of possible solutions.

Chapter 5 157

The comments of this last respondent suggest that a collection of solutions might better address the needs of a designer. The lack of solutions to design problems limits the guidelines’ helpfulness. A respondent wrote that the, “guidelines I follow are largely technical, relating to issues of accessibility. They serve more as restrictions than as tools.”

The comments regarding whether guidelines are helpful or not in design reflect on the process of design and the quality of the guidelines. The perception of helpfulness is based on how the guidelines are used (to validate, for example) or their quality (if they are poorly written, they can be of limited value). The authors of Research-based Web Design & Usability

Guidelines recognize that existing guideline sets are problematic and they argue that, “most web design guidelines are lacking key information needed to be effective,” (U.S. Department of

Health and Human Service, 2006, p. xv). Respondents in the study were asked to reflect on the quote regarding the effectiveness of guidelines in solving web interface design problems. While some respondents agreed that guidelines lacked key information, others disagreed and felt that that the design process was too complicated for guidelines to be effective, that guidelines contained too much detail, or that the notion of effectiveness in the quote was too subjective.

Table 5.4 illustrates the themes captured in the responses.

Chapter 5 158

Table 5.4: Themes captured in response to: A designer once said, “most web design guidelines are lacking key information needed to be effective.” Please share your opinions regarding the effectiveness of guidelines in solving web interface design problems.

Number of Percentage of Theme instances respondents Design is too complicated for guidelines to be effective 15 20.5% Guidelines are too simplistic 15 20.5% The notion of “effectiveness” is subjective 13 17.8% Guidelines contain too much detail 7 9.6% Technology change creates challenges for guidelines 6 8.2% Don’t agree with quote (for various reasons) 6 8.2% Guidelines are too restrictive to be effective 5 6.8% No further comment (beyond agreeing with statement) 6 8.2% Total Respondents 73 99.8% Number who skipped question 43

One fifth of respondents that the complexity of design made the task of writing guidelines which

addressed any design problem impossible. One respondent wrote, “At best they [guidelines]

solve basic challenges of design so designers can concentrate on solving more complex issues.”

Others suggested that guidelines were too simplistic. The comments of two respondents

represent the idea of guidelines as problematic due to their simplicity,

Web design guidelines can sometimes become too simplified and lack the technical

content that a designer might need to answer his/her specific questions. Often they

lack precise details which then defeat their purpose.

They are usually a high-level sentence or two and they are a blanket statement about

all situations. Not every situation/interaction is the same, has the same audience, or

Chapter 5 159

has exactly the same core task. Many guidelines don't provide you the “why”

something works or the level of detail needed to evaluate.

For these respondents, the complex variety of design problems appears to undermine the effectiveness of guidelines which attempt to address broader issues, as captured in the following response,

I think the issue with most design guidelines is that they try to be very general, so as

to be applicable to a wide audience. But this sort of defeats the purpose, as they

become too general for any one application.

Fifteen of the seventy-three respondents disagreed that guidelines tend to me too broadly focused. “I find them too detailed,” wrote one respondent about guidelines, a sentiment which was not an isolated example. Guideline detail was problematic in a manner similar to finding guidelines too simplistic in that the core issue was the inability to apply guidelines to specific design problems. When a respondent, in disagreeing with the statement, stated that, “guidelines only show details,” they suggest that design problems require a broader approach than guidelines allow. Unique design problems require detail. But with some guideline sets the details that are provided do not encompass enough use cases.

Other respondents (17.8%) moved beyond the debate of detail versus generalization, in order to consider the influence of context. Guidelines are intended to address design problems and are based on expert opinion or research which emerges from specific contexts. For example, guidelines written to address HTML version 2.0 (which was adopted as a standard by W3C in

Chapter 5 160

1995) may not be applicable to HTML version 4.0 (which was adopted four years later) or to

HTML 5.0 which is scheduled for adoption in 2012.

Guidelines are only as good as the context they were created for. Once you move

beyond that particular context (or time, in terms of web trends and innovation),

many elements become less trendy, less of a key design pattern, or less applicable.

Just like when a design style guide goes out of date.

The context of use is further complicated by the overlapping expertise at play in web interface design, where many fields or disciplines overlap as evidenced by the educational background of the study respondents. Web interface design is not represented by a coherent design space, but rather the confluence of disciplines, each with their own particular perspectives. As one respondent wrote,

There are many cross design disciplines which spill into doing web design, like for

example people who work in print or video, they design for the web with the same set of

eyes and make poor judgments in delivering a project for the web. The web needs some

basic technical knowhow which they could have neglected to study before approaching

the web, for instance, optimization of content, colours, sounds, video. Download file size

is another big problem with people who come from broadcast, and this is also from

companies and ad-agencies who step into interactive projects, the art directors and

creative directors ignore the limitations and say that it’s designed for broadband, killing

the user experience for many.

Chapter 5 161

In addition to the different perspectives introduced by those involved in the design process, is the

need to recognize change. Guidelines which addressed design problems a decade ago may no

longer be relevant today. Features once common on web sites, such as frames and even tables,

are less common now and have largely been replaced by CSS coding and updated HTML

specifications and web browsers. Solutions to design problems today may differ from those of

yesterday. One respondent wrote,

I think the challenge is that how we are designing for the web is rapidly changing

along [with] the technology and capabilities of the web. Therefore, web guidelines

can become outdated pretty fast. The best thing anyone could do would be to get a

web community (a wikipedia of sorts) happening with people sharing their

insights in an easy to access manner...

The speed of technological change and the potential obsolescence of specific guidelines was mentioned by 8.2% of the respondents. “Technology is moving forward in a fast pace,” wrote one respondent, “so the guidelines from yesterday might often not do the trick today.” The challenge is to recognize changes and ensure that designers address change, possibly through updating guideline sets,

Design guidelines from 10 years ago are already ancient! Expectations from

users change, technology limitations change and creativity grows exponentially

and so should your guidelines.

Chapter 5 162

Other respondents recognized the changing technological landscape, but still found fundamental principles in guidelines that could be applied even in changing circumstances.

I think guidelines are just that, opinions and directions on handling familiar

tasks, criteria, build methods, etc. As long as the guidelines are understood and

deviation is warranted, I do not believe they must be strictly followed, nor do I

believe guidelines necessarily encapsulate all circumstances or scenarios. The

guidelines, as the technology itself, continue to change and evolve over time,

though a fundamental set of guidelines persist.

The use of guidelines requires judgment on the part of the designer and a knowledge of when guidelines should – or should not – be applied. Some respondents cautioned that guidelines should not be considered in isolation. Instead, they should be viewed as part of a relationship between the guideline and the guideline user. If guidelines did indeed lack key information, as suggested by the authors of Research-based Web Design & Usability Guidelines, it was up to the designers to provide that information, to merge experience with advice. In the opinion of one respondent, the notion that guidelines should have key information was flawed, since, “they`re guidelines, not rules. Guidelines set you in the right direction. They don’t spell out the answer for you.”

The expertise of the designer using guidelines is a key factor in their value. If guidelines are tools to aid in design, the practitioner needs to know how to best make use of the tool,

Chapter 5 163

Guidelines are just that - guidelines. They sometimes require the mind of a

designer in order to implement them in a way that makes sense for the project. I

think it depends on the type of guidelines you're looking at. For something like

WCAG, the guidelines need to be very clear in order for them to benefit the

users relying on specific features / functionality. However, for brand-related

guidelines, there needs to be a bit of flexibility to allow the guidelines to fit the

project in a meaningful way.

Because the interpretation of guidelines was an issue in their use, one respondent was uncertain that guidelines were even the best tool to aid in the design process. “Design patterns are in many ways more practical than guidelines,” wrote the respondent, “because guidelines are often somewhat open to interpretation.” As noted earlier (see section 5.3.1) design patterns, or pattern libraries, were referenced by seven percent of respondents as the source of guidelines used in their work. The use of these libraries was further explored in a subsequent study (Chapter 6), where designers were interviewed and asked to discuss their use of guidelines.

Design problems emerge when there are multiple means to address a design. These choices, when mutually exclusive, are also known as trade-offs. The next section reports on how respondents answered questions regarding trade-offs in their work, and how they reflected on four examples of specific trade-offs.

5.3.4 Specific design trade-offs

A design trade-off represents a choice where the gaining of something results in the loss of something else – choices where one design aspect or element is literally traded for another.

Chapter 5 164

Trade-offs are the choices made between two valid design solutions. The literature references

four specific trade-offs with which respondents reported differing levels of awareness:

1. Ensuring site accessibility for most potential uses versus the use of graphics, images

and multimedia elements within a page (Regan, 2004)

2. Ensuring site accessibility while incorporating screen based controls or widgets, which

operate outside of browser based controls (Regan, 2004; W3C, 2008)

3. Optimizing the user experience while addressing the needs of content using either

scrolling or paging as a means of display (Norman, 1983)

4. Ensuring that the layout of a page allows users to find content they seek versus broader

content organization concerns (Veen, 2001)

Table 5.5 lists trade-offs and the percentage of respondents who indicated that they (i) had

experienced the trade-off, (ii) had not experienced the trade-off, or (iii) were uncertain about experiencing the trade-off.

Table 5.5: Respondent experience with specific trade‐offs (n=116)

Trade‐off Yes No Uncertain Skipped question Ensuring user accessibility vs. 65.2% 28.6% 6.3% 3.4% The use of images and multimedia Ensuring user accessibility vs. 31.5% 50.0% 18.5% 6.9% Employing screen based controls / widgets Optimizing user experience vs. 55.5% 33.6% 10.9% 5.2% Displaying image and text rich content Ensuring a web page is laid out clearly vs. 61.3% 31.5% 7.2% 4.3% Displaying dense or complex content

Chapter 5 165

Sixty-five percent of respondents experienced the trade-off between ensuring user accessibility and using images and multimedia. One of the main reasons cited was the use of Adobe Flash in the design of web pages. This browser plug-in is not ideal for ensuring accessibility, as described by one respondent,

More explicitly during the use of flash animations, came across this problem a

lot, especial whilst developing e-learning content which has a high focus on

accessibility, with all the best intentions in the world, some animations, videos

or interactive media can't possibly provide the same experience for sighted and

fully able users as it can to those that aren't. Instead the focus should be on

producing a relevant and engaging experience for both rather than a trade off.

Another respondent described how they were able to work around the trade-off,

It is important for service sites and portals to make accessibility a priority, and

for government portals, it is also a mandate. At the time we built this particular

site, Flash was not as accessible as it currently is, and we needed flash

elements to communicate certain features and interactions. Ultimately, we kept

the Flash elements in non-essential modules, so that primary functionality and

information delivery was not directly affected by the tangential Flash elements.

Even where a trade-off exists, there are ways to address user needs and to find creative solutions.

The trade-off between client demands and the designer's preferences is also evident in two

Chapter 5 166 comments, both of which place the blame for the trade-off on the client, “User accessibility has often been dismissed by clients more interested in a visual "wow" factor,” wrote one respondent, while another stated broadly that, “Some companies are not interested in ensuring accessibility.”

The only trade-off with which a majority of respondents indicated “No” experience was between ensuring user accessibility and employing screen based controls or widgets. “This is less of an issue than images/multimedia because most browsers handle these controls effectively now,” wrote one respondent, while another cautioned that as technology changes, so might designers’ experience with such trade-offs,

I'm starting now to run into some issues with web sites that will be both web-

and iPad-accessible. The problem is the extensive use of mouse hover effects

that are in widespread use across the web with CSS. While there appear to be

workarounds (double tap?) for the touch interfaces, it is clear our understanding

of hovers and how they are used in designing experiences will need to change.

Fifty-five percent of respondents reported experiencing a trade-off between optimizing user experience and displaying image and text rich content. Although a trade-off is posed as a binary choice, this is not necessarily the case. For example, a creative solution to a design problem may allow for both user optimization and the use of rich text, within a particular use context. “I display images and rich text to optimize the user experience,” wrote one respondent, arguing that the choices were not mutually exclusive. The respondents did not discuss trade-offs in isolation, but rather viewed them as existing within an overlapping array of design problems.

Chapter 5 167

Sometimes there are trade-offs, but sometimes displaying image and text rich

content improves the user experience for reasons other than efficiency. User

experience design has to also consider how much enjoyment a user gets out of

the experience, and depending on the target user it isn't always about how to get

where you want to go as quickly as possible. For our organization, we spend a

lot of time designing news interfaces, so we need to put the priority on

efficiency and usability. That said, Jim Carrey's website (not ours) is a good

example of a site created with a different kind of user experience in mind

(www.jimcarrey.com). While this site ignores many best practices for usability,

it gives its audience pleasure in exploring, providing for them an excellent user

experience. This keeps in mind the idea that the audience for this site isn't

coming there for critical news, show times or important information.

As with many facets of design, the interpretation of what constitutes a trade-off varies. While four specific trade-offs have been identified in the literature, the complexity of design results in a myriad number of trade-offs, which the next section considers.

5.3.5 General design trade-offs

“This is the stuff of design – making choices,” wrote one respondent when asked to describe experiences with trade-offs. Making choices is the very process of design. “Every design decision is a trade-off in some way,” wrote another respondent. “Most often a trade-off involved prioritizing something at the cost of something else.” It is the perceived “cost” that results in the notion of a trade-offs.

Chapter 5 168

Respondents reported that trade-offs are frequently a factor in their work. Figure 5.7

illustrates that 61% of respondents felt trade-offs occur “often” and only six percent reported that

trade-offs rarely or never occur.

Don’t know 3.4%

Never 1.7%

Rarely 4.3%

Sometimes 29.3%

Often 61.2%

0% 10% 20% 30% 40% 50% 60% 70%

Figure 5.7: Responses to question, “How often have you experienced web interface design trade‐offs in your work?” (n=116)

The nature of the trade-offs varied, although they can be grouped in five broad categories, as shown in Table 5.6. The highest number of trade-offs were a result of technical limitations faced by designers. In these instances, respondents referred to the inability of browsers, the servers, or other technical issues which prevented the execution of the design in a web-based environment.

Chapter 5 169

Table 5.6: Trade‐offs reported by respondents

Trade‐off description Number of Percentage of those respondents who answered this question* Designer preferences versus technical limitations 24 32% Designer preferences versus client demands 22 30% Usability versus aesthetics 13 17% Usability versus budget limitations 6 8% Other 10 13% Total 75 100% Did not reply to question 41 * In answering the question, “Please describe experiences you have had with web interface design trade‐offs in your own work,” some respondents included examples that referenced more than one type of trade‐off.

Trade-offs due to technical limitations emerged from general issues (“Sometimes the solution can't be applied for technical reasons.”) or more specific browser related problems, where “Most times, the design concepts ... are stripped down for the sake of load times or browser compatibility.” Other respondents concurred with the frustration surrounding the technical limitations imposed by browsers, “I could trot out another dozen examples of interface design trade-offs that wouldn't exist if supporting IE wasn't an issue.”

Demands by the client represented trade-offs described by almost one-third of the respondents. Clients (mainly external clients, although some respondents made reference to internal clients) placed demands on the project that forced designers to trade-off design aspects.

Usability issues played a role in 25% of the cases reported, as usability requirements conflicted with either design aesthetics (17%) or budget limitations (8%). One respondent reported that,

“designs often get dumbed down for the sake of usability.”

Trade-offs between preferences of the designer and their client were referenced with a hint of frustration, since the solution was not necessarily considered the strongest by the

Chapter 5 170

designer. Whereas technical limitations or budget restrictions can perhaps be appreciated, clients

demands can be problematic and the negotiation of trade-offs requires client education,

The trade off is between me and the client. It is a matter of negotiation and

education regarding how people will potentially use the Web site. Some clients are

very happy to negotiate and others are very reluctant. It also depends on where they

are in the Web site ownership process. Clients who are on their second, third,

fourth redesigns tend to want different kinds of information, such as user

preference data, rather than simple preferences of the Web design committee.

The need to educate the client is also referenced in the comment of another respondent,

The design process for web interfaces can often be a series of compromises

between best practice and client desires and goals. The education process

inherent in this can be an exhausting additional requirement to a project.

The frustrations of making trade-offs to address client demands are evident in the answers of other respondents, “making your client happy is a huge design trade off. Regardless of what the best solution is.” Another also suggested that clients might not have the best solution, “If the client wants something specific, it does not matter what the right way to do something is.” Trade- offs represent not only the need to choose between two options, but the need to recognize that something is lost in the choice – something is “traded off” in the process.

Chapter 5 171

5.4 Summary

That a majority of respondents used guidelines and over 80% thought guidelines were sometimes

or often helpful is not surprising in a survey that focused on guideline awareness and use.

However, the breadth of guidelines referenced by respondents was unexpected. Client-developed guidelines, company guidelines, pattern libraries, and accessibility guidelines were all referenced

by respondents. Guidelines are being used by the respondents, but there is no evidence of

homogeneity in the guideline sets used. When queried regarding awareness of five evidence-

based guideline sets, not one of the sets was used by a majority of respondents (47% of

respondents reported having applied Web Content Accessibility Guidelines, version 1.0). This suggests that the presence of research evidence may not be a determining factor in the choice of guidelines. Design knowledge, in the form of guidelines, is disseminated through a variety of sets, some evidence based, but most supported by expert opinion. The discussion of checklists or pattern libraries by respondents serves to expand the definition of a guideline set, beyond the template first introduced by Smith and Mosier (1986). Unlike the evidence-based guideline sets found in the literature, a checklist does not list references or sources, provide examples or a

detailed explanation of a potential solution to a design problem. Similarly, a pattern library is a

shared knowledge repository different from guidelines. A narrow definition of a guideline set is

subverted by the description of guidelines presented by the respondents. Guidelines are

perceived in very broad terms. This implies that while evidence-based guidelines are arguably

the strongest link between published research and design advice, the format of the advice may

have a greater impact on its adoption than the presence of supporting research evidence.

Regardless of the definition, the use of guidelines by respondents is dependent on three

factors: context of use, the ability of the designer to interpret and adapt the guideline

Chapter 5 172

recommendations, and the quality of the guidelines themselves. Respondents considered the use

of guidelines in contexts that differ from those in which they were developed as problematic.

Perhaps this is to be expected. But the context from which a guideline set emerged is not always

clear. The Research-based Web Design & Usability Guidelines (U.S. Department of Health and

Human Services, 2006) may have been developed predominantly for use in health information sites, but not only is the supporting evidence drawn from outside the field of health care, the authors explicitly address a broader audience. Would designers reject such a guideline set if they were not working within health informatics? Comments in the survey suggest that this might be the case, as the importance of context of use was referenced by numerous respondents.

Interpretation was recognized by some respondents as a problem when using guidelines.

Respondents did not recommend any specific solutions to this problem, with the exception of

one respondent who suggested not using guidelines at all.

The quality of guidelines was also recognized as a factor in their usefulness. Respondents felt that guidelines recommending restrictive solutions, or solutions with the potential for only narrow application, negatively influenced their usefulness. While some respondents suggested the guidelines needed to be broader in terms of focus and examples, other respondents suggested the opposite, that broad guidelines were too abstract and did not provide specific solutions to design problems. These two positions are less contradictory than they would appear. Both opinions speak to the need for clear assistance when facing design problems. In some cases design problems can be narrowly defined, whereas in other instances design problems are broader in scope. Ideally guidelines should address both cases.

Guidelines are used in various stages of the design process. Respondents suggested considering guidelines prior to starting a project, when design ideas were being formulated.

Some respondents felt guidelines were good to use during the entire process of design, while

Chapter 5 173

others felt they were most useful once the design was almost completed, as a means to check if the design had missed any important considerations. Such use implies guidelines functioning as

reminders or memory aids. Finally, respondents also suggested utilizing guidelines only in

support of designs choices when presented to clients or peers.

Design trade-offs were a factor in the design work of many respondents. Trade-offs were

recognized by respondents as playing a role in the design process. Design trade-offs are currently

not well addressed in the literature, yet the subject of trade-offs elicited strong opinions from

respondents from differing perspectives. For some respondents, trade-offs were a problematic

(and impairing) factor in design; others indicated that design itself was the choosing of mutually

exclusive solutions and thus trade-offs were not problematic.

The survey data showed that guidelines have a role in the design process, although the

specific role varied greatly. The study reinforced the notion that interpretation was an important

factor in the use of guidelines, as was the use context and the quality of guidelines.

Respondents were asked if they would be available to take part in a telephone interview.

The interviews form the basis of a subsequent study, where the use of guidelines, the prevalence

of trade-offs and knowledge sharing were considered in more detail. The results of this study are

presented in the next chapter.

Chapter 5 174

Chapter 6

Guideline use, design trade‐offs and knowledge sharing (Study #4)

6.1 Introduction

The survey of web designers described in Chapter five concluded with an invitation to participate in an interview. The interviews with twenty professional web interface designers allowed for a focused discussion of issues that were revealed in the survey and to further explore the opinion of designers.

6.2 Methodology

Web interface designers who had participated in the web-based survey and expressed interest in further participation were contacted to take part in a phone interview. The data collected in Study

#3 from these participants was used to inform the interviews (by modifying the questions asked to reflect survey answers). While this method allowed for the clarification of specific answers provided by participants in Study #3, the interviews also addressed topics that were not addressed in the web survey. An interview script (see Appendix J) was used to guide the discussion. Participants spent from 25 to 45 minutes discussing web interface guidelines, design trade-offs and knowledge sharing.

Chapter 6 175 6.2.1 Method

The aim of the interviews was to explore in greater detail (compared to Study #3) the opinions

web interface designers held regarding interface design guidelines, trade-offs and how

knowledge is shared between practitioners. The interview format also allowed for leeway in

considering a specific opinion or idea expressed by the participant. A phone interview was

selected as the method for interacting with the study population due to the geographic dispersion of potential participants and to allow for more flexible participant scheduling.

6.2.2. Apparatus

The interview script was divided into five sections; an introduction which included a reiteration of study confidentiality and participant consent, questions regarding the use of web interface

design guidelines, questions about design trade-offs, questions about guideline awareness and

knowledge sharing, and a concluding section which asked participants if they had any further

thoughts or ideas regarding the use of the guidelines or any related topic not covered in the

interview.

A beta version of the interview script was tested with three individuals; a web interface

designer (who represented potential participants) and two individuals who had assisted with the

development of the web-based survey tool and were, like potential participants, familiar with

questions asked prior to their involvement in the interviews. One mock interview was done in

person, whereas two were conducted by telephone.

Skype was used for the interviews, as participants were geographically dispersed

(potential participants were located in Canada, the U.S., and India). The use of Skype allowed for

Chapter 6 176 cost control, but also the use of a third party recording device. The recording software called

“Pretty May” was chosen to record the conversations following tests of a number of recording

applications for quality, ease of use and features. Digital files produced during the course of the

interviews were transcribed and comments which would either identify participants or their

specific clients were edited for confidentiality1.

6.2.3 Participants

Participants were recruited from the 116 professional designers who participated in the web- based survey. Thirty individuals agreed to take part in the study, but problems with scheduling opportunities reduced the number of participants to 20. Participants were not compensated for participating in the study. Since the participants all completed the web survey (Study #3), demographic data was collected prior to the interviews. Table 6.1 lists the years of web interface design experience, the size of the company employing the participant, the number of web designers employed by the company in addition to the participant, age range, gender and country of residence.

1 For example, references to specific companies were changed to euphemisms such as “large computer services company” or “Client A”.

Chapter 6 177 Table 6.1 Interview participant demographics

# Web interface Company size # of web designers at Age range Gender Country of design experience company in addition residence to participant 1 Over 25 years Self‐employed N/A Over 55 Male U.S.A. 2 16‐20 years 2‐10 1 45‐55 Male Canada 3 11‐15 years Over 1000 15 + 45‐55 Male Canada 4 11‐15 years Over 1000 15 + 35‐44 Female U.S.A. 5 11‐15 years Over 1000 15 + 35‐44 Male Canada 6 11‐15 years 101 to 500 5‐9 Over 55 Female Canada 7 11‐15 years 11 to 50 10‐14 35‐44 Male Canada 8 11‐15 years 11 to 50 1 35‐44 Female U.S.A. 9 11‐15 years 2 to 10 2‐4 45‐55 Male Canada 10 11‐15 years Self‐employed N/A 45‐55 Male Canada 11 11‐15 years Self‐employed N/A Over 55 Male U.S.A. 12 11‐15 years Self‐employed N/A 35‐44 Male Canada 13 11‐15 years Self‐employed N/A 35‐44 Male Canada 14 7‐10 years Self‐employed N/A 25‐34 Male U.S.A. 15 4‐6 years 51 to 100 5‐9 35‐44 Male Canada 16 4‐6 years 11 to 50 2‐4 25‐34 Male Canada 17 4‐6 years Self‐employed N/A Under 25 Female Canada 18 1‐3 years Over 1000 1 Under 25 Male Canada 19 1‐3 years 11‐50 2‐4 Under 25 Male India 20 1‐3 years Self‐employed N/A Over 55 Male Canada

Sixty-five percent of the participants interviewed reported at least 11 years experience as web interface designers. In terms of education, nine of the participants hold graduate degrees, while three described general studies but did not report specific degrees.

Chapter 6 178 6.2.4 Procedure

Participants were contacted by email in order to schedule an interview time and to provide them with a digital information and consent document (see Appendix K). They were asked to review the documents and reply with consent to participate. Participants were contacted according to the mutually agreed upon schedule. The interview began with a brief description of the study, a review of the consent document and a reminder that participants could withdraw from the study at any time. Verbal consent was provided by the participants (in addition to email consent provided prior to the start of the interview), as was permission to record the conversation.

Participants were reminded that information that might personally identify them would be removed from the transcriptions. Such information included names of individuals, companies and clients.

6.2.5 Measures

The interview script included 23 questions. Following a general question regarding the participant’s role in a recent design project, there were 11 questions regarding guidelines, five questions related to design trade-offs, four questions regarding knowledge sharing and two closing questions where participants were asked if they had any further thoughts on guidelines specifically, or on any topic that may have emerged in the course of the interview. Questions asked of each participant were similar, but not identical since their survey answers affected the interview script.

Chapter 6 179 6.3 Results

The interviews covered three broad areas; guideline use, design trade-offs, and knowledge

sharing. Data from these areas will be presented first. In addition, three areas of focus which

emerged while conducting interviews will be presented; (i) pattern libraries as guidelines, (ii)

preference for expert opinion or research, and (iii) solutions to design trade-offs through

guidelines. The chapter will conclude with a summary discussion.

6.3.1 Use of design guidelines

Each interview began with a general question regarding the role the interviewee played in their

most recent project. When asked “Did you use any web design guidelines in your most recent

project?” 15 interviewees indicated use of guidelines. Of the five that did not, two said that they used knowledge gained from a familiarity with guidelines, although specific guidelines were not consulted during their most recent project.

The guideline sets used in the work of the participants varied by type, confirming

findings from the survey. Guidelines supported by expert opinion (client provided) were

mentioned as frequently as those supported by research evidence. There was no guideline set that

all participants applied to their work, although five of the participants made specific reference to

using the WCAG accessibility guidelines. The discussion with participants revealed no single

guideline set that was applicable to their work, but rather multiple sets. The interviews revealed

that guidelines and guideline sets were combined as needed. In one example, a participant

discussed using WCAG guidelines, heuristics, and client provided guidelines to help solve a

single design problem.

Chapter 6 180 The complexity of some design problems results in the necessity to utilize multiple

guideline sets in the search for a solution. One participant discussed the pressures affecting the practice of design; client needs, the unique nature of each project, the need to improvise and react to changing demands,

We use a combination of our internal guidelines and the codified knowledge out there.

It’s difficult to know when you start a project which you are going to use. It’s only when

you get in there and see a problem and wonder if it’s been solved before, or if a solution

is in a pattern library or are there best practices which address this or should we do some

research into this area or are there corporate guidelines you’re mandated to use.

Each design project has unique challenges, so sources for solutions to these challenges vary.

Guidelines from clients helped to expedite the design process. A participant discussed the positive aspect of having guidelines provided by a client who had,

… a robust website and any design issue you worked on had a library which would have

a huge set of guidelines. You could almost cobble together a website just following those

instructions. It’s very helpful in that the guidelines can help you solve problems.

Saving time solving design issues is positive for both the client and the designer. For example,

“They [the client] have spent countless hours developing a video player, so you don’t have to worry about that, just come in and use it.” Such design work is modular in nature, but allows the design team to focus on a smaller number of problems.

Chapter 6 181 A guideline set not only cannot cover every potential design problem, but the interplay of guidelines can be problematic. Even when guidelines do not contradict each other, they may not

complement each other in a manner that creates a strong design. “There were things that it was

really helpful with because the client had solved a lot of UI problems that allowed us to focus on

other ones,” shared the participant, before considering the downside. As one interviewee

commented,

The negative to that is this whole thing that you hit up against things where what we were

doing doesn’t work within a certain requirement so what do we do? Or we would comply

with all the guidelines and go, wow, this is Frankenstein. Because everything had been

reduced down to these objects and you’d put it together and realize there is no soul to this

thing.

Previous studies have argued that the higher order interaction of guidelines potentially affects the design in ways that are too complex to comprehend (Cherry, Muter & Szigeti, 2006.) Interaction of guidelines with each other may create unexpected results. One way that the threat of

“Frankenstein” sites can be addressed, is through the use of pattern libraries, which are different than guidelines. They are knowledge repositories of a different nature; not compiled from abstracting a design problem, but rather from compiling solutions to specific problems. Pattern libraries were referenced in half the interviews. These libraries did not offer guidelines in the form of objects described in the quote above, but rather provided the designer or design team with various solutions to problems.

Chapter 6 182 Interview Focus: pattern libraries as guidelines

The consideration of pattern libraries as guidelines emerged in Study #3. Survey respondents had included pattern libraries in their list of guidelines, and the interviews allowed for a specific discussion of these collections of design solutions. As defined in Section 1.2, a guideline is a principle or criterion guiding or directing action, and pattern libraries were viewed by practitioners as such; they helped to direct action. “The pattern libraries are great for showing that there is a pattern here that you should acknowledge and use and benefit from.” Pattern libraries like the Interaction Design Pattern Library at Welie.com are key resources for some developers. “The Welie.com patterns,” said one participant. “That is my first and most important resource. If I don’t find it there, there is the Yahoo pattern interface library.”

These libraries are used in the manner of guidelines; to address specific design problems.

The pattern libraries offer the designer various solutions to problems which might be broad (such as designing e-commerce applications) or narrow in focus. “I use them usually when I’ve got a specific problem,” said one participant, and then continued to elaborate using examples of such problems. Pattern libraries represent a, “way to deal with a navigation issue or some complex data that needs to be easily filtered and I’m not quite sure what the method would be. So I use the Welie patterns, which have different options and many use cases for a particular treatment; a table filter, carousal or a list builder.” The pattern libraries represent a method to communicate a best practice, one that emerges from the practice of design. These libraries are not used as a means to broadly define design issues in a project, but for specific issues. “I just don’t think of using them comprehensively as my overall approach to the site,” cautioned one participant.

Chapter 6 183 6.3.2 Factors in choosing guidelines and/or guideline sets

The survey in Study #3 revealed how guidelines helped in the design process, but not the reason

for choosing a specific guideline set, which was explored in the interviews. Whether a guideline

or guideline set was adopted by a designer depended on four factors; (i) the timeliness or recency

of the guideline, (ii) the trustworthiness of the guideline source, (iii) the rationale or evidence

supporting the guideline and (iv) if there was a strong match between the guideline and the experience of the designer.

Timeliness of guideline

The rate of technology change influences web interface design. Research which focused on the performance of a particular system may not be valid as the system changes. For example, guideline advice based on 56Kb modems may be less relevant when the user base is predominantly broadband. Not only does technology change, but so do the habits of users. How a user interacts with different input devices (such as touch screens or voice activation) changes in a manner that potentially makes some guidelines not only irrelevant, but possibly incorrect. The rate of both technological changes and changes in user behaviour was identified by participants

as a factor in whether a specific guideline was adopted. One participant felt that if a guideline was over two years old, they would be inclined to mistrust the advice. Given a choice, the more current guideline would be viewed more favourably. Another participant discussed the need for

ensuring that the evidence informing guidelines is constantly updated in order to maintain credibility. While jokingly blaming Jakob Nielsen for disputes with clients, the point made is relevant to the discussion of guideline timeliness,

Chapter 6 184 He [Jakob Nielsen] is to blame for my fights with clients over scrolling pages. Because of

something he wrote in 1996, I still have “above the fold” arguments with clients, which is

a great example of a guideline that is still hung out there in the face of contrary evidence.

If you were to still be using dated medical guidelines, we’d be leeching people (laughter).

Trustworthiness of guideline source

Whether a guideline is drawn from research evidence or practice, the source must be trusted by practitioners. In this context, trust is defined as confidence in the competence of the guideline author(s). The rationales for judging competence may be varied, but many participants made reference during discussions to the importance of source credibility. One participant discussed the importance of credibility in assigning trust,

If the [guideline] is done by somebody I think is credible I take it carte blanche because

who among us has time to research all of these things themselves. So if I trust that

person’s professional judgment then I just take what they have to say and use it. I would

prefer professional advice provided that I trust the individual. If I receive advice from

someone, and I’ve seen their work before, then I know that they understand what I’m

trying to do and that the solutions they propose in their work and the work that they do is

similar to mine and of a quality that I like and is by nature alike then it’s a question of

pedigree.

Conditions for bestowing trust differed by individual. What constituted “pedigree” for the participant above was unclear, but in the discussions with participants it emerged that the criteria

Chapter 6 185 for allocating trust was personal. For example, two participants trusted the advice of a guideline

due to the endorsement of recognized institutions (Yale and M.I.T. in the case of one participant

and W3C in the case of the participant quoted below),

The reason that I hold the W3C guidelines to such high esteem is that the W3C is a

regulatory body for the web and know better than anyone else how it operates and

they’ve devoted their entire existence to essentially “the web”. So I feel that given the

proposition that these guidelines will help sites to be more accessible and technically

correct and knowing who they’re coming from, I find that easier to swallow.

Participants also identified the endorsement of guidelines by friends or colleagues as a source of trust. Guidelines recommended by individuals carry more weight in the eyes of some participants,

If my friends or industry people that I respect are using it or recommending it, then it

would have more weighted value than some guideline that’s been published that no one

has ever heard of or a friend tells me is full of errors. So it’s mostly crowd sourced; my

friends like it, my associates like it, so I should probably take a look at it.

Crowd sourcing, in this context, represents a perspective or opinion shared by a larger group.

Where there is a consistency of opinion, trust may emerge. The influence of the group was

evident in the discussion of other participants as well,

Chapter 6 186 ... the evaluation may come from other people in the industry or related industry who say

I read this, does this make sense to you, and then I take the temperature out there. So I’m

finding my perspective is shared by other people, or we’re in agreement that [a particular

guideline] doesn’t make any sense.

In this context, trust is external to the guideline. It is not the information contained within the guideline itself which bestows trust, but rather the guideline is trusted due to an external source.

This is not to suggest that the content of the guideline itself does not play a role in its adoption.

When asked about the factors influencing the adoption or use of a guideline, the rationale or argument presented within the guideline itself was identified by over half the participants.

Rationale or evidence supporting the guideline

A guideline has either implicit or explicit rationales for use. Regardless of whether a guideline is based on expert opinion or research evidence, all guidelines include arguments for their adoption. The strength of this argument is a factor in guideline use. Regarding the rationale for adopting a guideline, one participant said that,

I would expect to see some depth of reasoning there, beyond saying this has come up a

few times and here is my conclusion. Something beyond anecdotal and personal

experience has to enter into it. Something more robust. Or I expect it to be qualified with

limitations acknowledged. I want to see someone taking the time to go into depth before

they say this is something I’m going to get behind and really advocate in 80% of the

cases.

Chapter 6 187

The “someone” in this case is the guideline author, as opposed to an external source. The guideline must reference reasonable sources, although the sources do not have to be research based. Regarding the difference between expert opinion and research evidence, “I would be inclined to evaluate the information of both sources,” said one participant, “and see how I feel about either one of them as a measure of its validity, rather than categorically saying one is less valid than another.”

The most referenced reason for adopting a guideline was the expertise of the designer themselves. If the guideline supported or reinforced the expert opinion of the designer, then they were more prone to use the guideline. In other words, if the solution posed by the guideline was one that resonated with the designer’s experience, then the guideline would be used.

Consistency between guideline and design experience

Expert opinion is a term used to describe the ideas of expert practitioners. The participants in this study, based on their work experience and backgrounds, could (with the exception of three participants who reported between 1 to 3 years experience) be considered experts. Their experiences as practitioners (Table 6.1 shows that 13 of the 20 participants reported over 10 years experience as web interface designers) are arguably an important factor in their attitude towards and potential use of a guideline. Their design experience is crucial, since their opinions have been formed by years of solving design problems. One participant called his use of experience an instinct, “I think I’ve got instinct,” he said. “You go by your past experience and your feel for it.” If the guideline advice matches the past experience of the designer, they would be more inclined to accept the advice,

Chapter 6 188 I would look at a guideline and wonder if it rings with my experience and if it didn’t, I’d

be reluctant to use it. However, if a client came up to me and says the research makes

sense then I would use it. If it has good provenance, good research behind it, then it

wouldn’t matter if I liked it or not. I could live with it no problem.

For other participants, the provenance of the guideline or the presence of research is less of a concern than the perceived credibility of the advice. During one discussion, it emerged that even anecdotal advice could be adopted. “Anecdotal information may be accurate, if it sounds credible to me,” said a participant, “and that’s based on my being in the business over 35 years. If it sounds credible that means it matches some of my knowledge and expertise and makes me think that it sounds reasonable.”

These factors; the recency of the guideline, the trustworthiness of elements both external and internal to the guideline and the match between guideline advice and experience, all play a role in the potential use of a guideline. These three factors are in addition to a fundamental consideration that participants referenced; context. Whether a guideline applies or not will depend on its relevance to a specific design problem. In describing the potential adoption of a guideline, a participant articulated three steps that they would take,

First is appropriateness to the situation. There may be a guideline that in one way applies

to what I’m working on but because of some extenuating circumstances it simply will

not…is not really valid. Understanding the context is the first thing that helps me

understand whether or not it is applicable.

Chapter 6 189 The second thing that helps me understand whether or not to use it is, how well

thought out and how well designed the idea seems to be and that is based on my own

internal clock of what I believe design to be.

And a third would be to what extent I know that they have done any research or

can back it up with real world experience.

Interview Focus: preference for expert opinion or research

Study #3 found awareness and use of guidelines supported by research evidence lower than

expected. During the interview, participants were asked if they believed “there is a difference in

the validity of either type (expert opinion or published research)” and why they held this belief.

Thirteen of the participants said that there was no difference in either source, and that research evidence was not necessarily a factor in their decision to use a particular guideline set. Of the remaining seven participants, three favoured guideline sets supported by expert opinion and four favoured guideline sets supported by research evidence.

“I think they [expert opinion or research evidence] are equally valid,” said one participant, “because they are coming from different viewpoints. To some extent, I always wonder with scientific research, especially when you’re trying to judge someone’s experience on

a website, there is no way you’re going to introduce bias in there, it’s just impossible not to.”

6.3.3 When guidelines are used in the design process

Participants were asked to “elaborate on where in the design process [they] used guidelines” in

order to gain a better understanding of the role guidelines might play in the design process. This

question was not explored in the survey, although the first study (see Chapter 3) revealed the use

Chapter 6 190 of guidelines in different stages of the design process.

The specific number of steps in the process and their description differs by author

(Preece, 2002; Shniederman & Plaisant, 2009), but the process encompasses the following actions; information and requirements gathering, project planning, design, development, evaluation and testing, and delivery and client approval. The design process can be linear or iterative, and can also involve branching and other complicating factors. Guideline sets such as the Research-based Web Design & Usability Guidelines (U.S. Department of Health and Human

Services, 2006) include guidelines specific to different design phases, since research exists of potential use during each step of design.

When asked where they would use guidelines in the design process, answers varied from early, middle of the process, late or across every step. Table 6.2 lists the process actions and the temporal relationship.

Table 6.2: The web interface design process

Temporal Action

1. Information and requirements gathering Early 2. Project planning

3. Design Middle 4. Development

5. Evaluation and testing Late 6. Delivery and client approval

Chapter 6 191 Early in the process

Practitioners who discussed using guidelines at the inception of a project stressed the importance of establishing the design scope prior to beginning design. “They are definitely something that are in there at the start,” said one participant regarding guidelines. “The first place you start to apply guidelines is when you gather an understanding of what the users requirements are.”

Another participant elaborated on the need to reference guidelines early in the process in order to define the project scope,

We certainly use them early on when people are gathering requirements and preconceiving

how they want this thing to look and we sometimes use that to sometimes temper the

tendency to design before the requirements are gathered. When we’re at the design phase,

it’s crucial that we can refer to the design guidelines to limit design decisions.

Once the requirements and project scope are identified, the design phase begins. To some practitioners, guidelines have greater value at this stage.

Middle of the process

The start of a project does not necessarily involve design problems, and some practitioners have processes in place that do not require reference to guidelines. Needs assessment and requirement gathering are processes that can be relatively consistent from one project to another. As one participant described, guidelines are not needed when there is an existing work process where not only requirements are gathered, but potential design problems are identified. This allows for guidelines to be consulted in the design phase.

Another participant described a preference to rely on experience (instead of guidelines) at

Chapter 6 192 the start of a project. In sketching out possible approaches, potential design problems begin to

emerge,

I would say [I consult guidelines] more in the middle. In the early stages when you’re just

doing very rough sketches I don’t try to think about whether this fits into a, b, c or d

guideline. When I start, I’m trying to create an optimal experience for the user, so my

sketches are a combination of trying to understand for myself what the problem is and

also what the solution might be. And then as I get a better handle on what that solution is,

I start moving towards, when appropriate, trying to look for guidelines and standards.

There are differences between a designer working in isolation and working in teams. When the design process involves numerous designers, a need exists for consistency across the work – which is another area where guidelines might help. When asked if there is a difference in approach when working within the context of a team, a participant replied,

Of course there is, and that’s when I think website guidelines are the most important

because you can have five designers doing really nice designs that are completely

inconsistent with one another. So I think design guidelines become more important the

larger the number of designers that are working on something.

End of the process

Study #1 presented results where some participants felt guidelines were of most value at the end

of the design process, as a means to justify design decisions made in the absence of guidelines.

Chapter 6 193 The notion of validating decisions is echoed in the discussion with an interviewee who felt the end of the design process, once there was an actual design to consider, was the best point to consult guidelines,

More of a validation check in the latter stages of a project. It’s been my experience that

the assessment of the project user interface against industry guidelines are done after

prototyping and user validation.

Interviewees who identified specific points in the design process for guideline use did not discount their use elsewhere, but rather identified certain phases where guideline referral was most beneficial.

Throughout all stages of design

Since some guideline sets cover all phases of the design process, it is not surprising that some participants discussed the need for guidelines throughout. “It doesn’t matter where you are in the process,” said one participant, referring to guidelines as standards. “If you are doing something that can be answered by what’s inside those standards then you follow what’s inside those standards.” Participants who discussed the need for guidelines throughout the design process made reference to the numerous decisions that had to be made throughout, from needs assessment, to design, to evaluation and even post-delivery in the form of ongoing maintenance.

Chapter 6 194 6.3.4 Design trade-offs

The notion that design involves making choices and that guidelines might assist designers in

making a correct choice, suggests the presence of trade-offs. The next section reports data

collected in the interviews with participants about the nature of trade-offs in their work and

whether guidelines are a means to help resolve difficult choices. The interviewees were asked,

“It has been suggested that interface design involves a series of design trade-offs. Trade-offs

represent choices a designer must make between different design solutions. Are the design trade-

offs you face unique to each project, or do you find similar trade-offs occur in various projects?

Please elaborate.” As revealed in the survey, most participants faced trade-offs in their work, but

the frequency and nature of the trade-offs varied. During the discussion of this topic, the

frustration of the participants was clear in the tone of their voices. In trade-offs something is lost

in the act of making a choice. “Every decision you make closes the door to other things,” said one participant. “The minute someone opens their mouth on a blank page, a trade-off has been

made. As you’re working, I think good designers understand the trade-offs they’re making and

can prioritize them.”

The need to prioritize the choices facing the designer emerged in numerous interviews.

While, in the words of one participant, “trade-offs are constant,” the successful resolution of

trade-offs required, “the designer to consider different points of view and to make trade-offs in

terms of priorities.” Participants discussed three types of trade-offs; (i) timeline and budget

conflicts, (ii) client related demands, and (iii) web clarity versus the density of content.

Chapter 6 195 Timelines and budgets

Project management literature argues for the importance of establishing a clear scope for a project in order to meet the challenges of deadlines and budgets (Gido & Clements, 2009; Grey

& Larson, 2006). The interplay between scope, timeline and budget involves trade-offs, and the discussion with one participant revealed this to be the most common trade-off. “That happens a lot,” said the participant. “With the companies I’ve been working for, a lot of times we have to give up on ideas because of development time, or money wise, or time wise. I think these are the most common trade-offs that I’ve been exposed to.” This specific trade-off is not unique to web interface design, or even to a broad design context. Similarly, trade-offs between the wishes of clients and practitioners are not unique to web interface design. As one participant said, “There is the time and money trade-off, we could do fantastic things if it wasn’t for clients and their constraints.”

Client related trade-offs

Client related trade-offs were identified in the findings of the survey in Study #3. The interviews allowed for a discussion of the nature of this trade-off. Clients, in the eyes of interview participants, did not necessarily help to resolve design issues and might in fact be detrimental to finding a solution,

The kinds of trade-offs that I experience are usually the result of the client not being

aware of any of the guidelines or standards. I spend an inordinate time educating my

clients about all kinds of guidelines throughout the work. I mean design guidelines,

writing guidelines, instructional guidelines, technical guidelines… and quite often the

Chapter 6 196 reason I am being hired by a customer is that they don’t have the in-house knowledge and

capability to do this in-house so they are contracting it out. That said, even after I educate

someone I tend to do that on the fly, so as we go through the iterative process I will

include something in the writing of the copy or the storyboard and it’ll come back with a

question of “why did you do this?” or “I don’t like that” and I will respond with “well,

here is the guideline or the trend or convention that is currently used…”, and hope that

the customer accepts my explanation. When they don’t, for whatever reason, that’s where

the trade-off occurs.

Trade-offs involving clients appeared to be the most frustrating for practitioners. The frustration grew from the conflict of expert and non-expert. “Most of the time the trade-offs involve the way someone who doesn’t know how to design wants something,” said one participant. The frustration was further compounded by the inability of guidelines to help, “what I find is you can have the most concrete set of guidelines and standards and they don’t matter at all to someone who is a decision maker.”

Two participants discussed specific problems with clients that involved trade-offs; they were related to how much content was displayed on the screen and how it was displayed. This is a trade-off identified in literature, involving web clarity and the density of content on a page.

“Trade-offs tend to come where clients want to add more to the screen. How do you guide them through that mine field, [explaining] that these decisions are detrimental to their end goal and not solving the problem that they think it is?” The participant described a scenario where they were forced to trade-off clarity or accessibility for the demands of a client. “You need a nuanced discussion where it’s known that if you go down this road, you can’t go down another road and

Chapter 6 197 we’re all ok with it. That’s how I look at trade-offs to pursue one road and not another and being ok with it.”

To be “ok” with a trade-off comes from experience. Discussions with participants often

touched on the adversarial nature of some trade-offs. On the one hand, the experience of a designer in solving a problem leads to frustrations if the solution runs counter to a client need

since the designer was hired due to their particular expertise. Yet on the other hand, with

experience comes the recognition that as long as a client clearly understands the repercussions of a trade-off, it is one less design decision that requires attention.

The trade-off between clearly presenting information and client demands for pages with

dense content is not easily resolved. “The clients sometimes want to add too much information

and overwhelm the user. How do you design for elegance? How do you not bombard them with

choices?” asked one participant, a sentiment echoed in other discussions regarding this particular

trade-off.

Trade-off between web clarity and presenting dense content

The tension between simple, clear design and the challenges of presenting rich content which

presents more information to the user emerged in a number of interviews. Although this trade-off

is arguably more narrowly defined than the broader issues surrounding client needs, it was

mentioned in the discussion of seven participants. “In terms of trade-offs,” mentioned one

participant, “there are some things that run throughout [projects]; the split between volume of

data and information appearing on a screen versus making that data and functionality consumable, so that people do not feel overwhelmed, that seems like a consistent trade-off and certainly in my world that is something we constantly struggle with regardless of the

Chapter 6 198 application.”

In the opinion of one participant, such trade-offs are not infrequent, “trade-offs at the

level of information display, of reducing complication at the level of the interface, are trade-offs

being made constantly.” There was some hope that guidelines could help resolve trade-offs and

provide designers with the necessary information to make correct choices.

Interview Focus: solutions to design trade-offs through guidelines

Interviewees were asked, “In your experience as a designer, do you believe guidelines would

help you find solutions to design trade-offs?” While one participant was clear that guidelines do

not help resolve conflict with clients, other participants described positive experiences using

guidelines. The W3C accessibility guidelines were referenced in this context by three participants who said that the risk of contravening such guidelines helped to direct discussion

when seeking solutions. A client might demand a certain approach, but the guidelines have been

used successfully to dissuade.

“Such third party support can be helpful,” said one participant.” I use guidelines when

they are convenient for me, sometimes to make a point, sometimes to move a client off a point. I

think they can absolutely be helpful. They can also take design out of your hand when they state

that a button has to be this and not that.” The notion of support is helpful, because guidelines do

not always offer a solution, but rather either options or clarity as to whether a particular solution

is correct or not. One participant discussed how the use of guidelines in design work helps

identify potential trade-offs and problems,

Chapter 6 199 I think the style guides are most useful in pointing out the problems and telling you what

people physically can and cannot do. Yes, there are some solutions like if you want a

huge document they would tell you give them a PDF printable which you may not have

thought of yourself. So there are some solutions but I guess what I found them most

useful in telling me the problems that exist and then you have to work on solving them if

you can.

6.3.5 Knowledge Sharing

The survey in Study #3 found that over half the respondents reported that they learned about web design guideline sets via blogs or other designers. The interviews explored broader knowledge sharing, not just specific to guideline sets. Participants were asked to, “Please describe how you share web design knowledge with other designers.” Knowledge sharing assumes both the seeking and dissemination of information, particularly design solutions. Table 6.3 shows the frequency with which various methods of knowledge sharing were mentioned during the interviews.

Chapter 6 200 Table 6.3: References to knowledge sharing between designers

Method of knowledge sharing References in Percentage interviews of total Face‐to‐face 11 42.3% Email 4 15.4% Blog posting 3 11.6% Digital services (wikis, listservs, content 3 11.6% management systems None 3 11.6% Phone 2 7.7% Total 26 100.2%

The process of knowledge sharing was described as largely informal. The most common means referenced was meeting face-to-face. “When I work in a team, I share something new that I found, or a new idea that I’m not 100% sure that it’s going to work,” said one participant, “so

[face-to-face] I can ask them about it to have a look and see if they like it or not.”

“Sometimes you go for drinks with friends and we look through each other’s sketchbooks,” commented one participant, illustrating the informal nature of sharing. Using electronic communication is another means to share information. “Sometimes I see them face-to- face and sometimes I send them a screen capture.” The choice to communicate in person or via electronic means appears to be a matter of personal choice or a result of the proximity of the participant to other designers. The use of electronic tools to share knowledge was described as intuitive, since participants spent most of their day at their computers. Electronic means included email, instant messaging systems or blogs. “Sometimes communication is done electronically,” described one participant, “through e-mail or IM. Sometimes I would send screenshots or tell them where I posted a doc on our intranet site.”

Over half the interviewees suggested that most design solutions were found without

Chapter 6 201 consulting other designers. “I’m an independent consultant,” said one participant, “and very often I’m the only user designer in the whole company that has hired me. So speaking to others about patterns and so forth…is like being the only doctor in a wilderness situation….there is no other doctor to consult.” Another participants discussed how they felt sharing information was atypical,

I find sharing information to be lacking. For me, design tends to be a largely solitary

activity so I don’t… maybe there are a couple of people I would email and ask “what do

you think of this? I’ve hit a wall and don’t know where to go.” But largely it’s been

solitary for me.

One participant bemoaned the lack of sharing, uncertain why the community does not interact more often, “All of people who do this stuff are lone guns. When we do get together, it’s always easy to get together and talk, and if we can find the time and room it’s great to do. There is no society of internet professionals that is that successful.”

6.3.6 Guideline format

The survey revealed that designers use a number of guideline sets in their work, but it was unclear if the format of the guideline was a factor in their use or if certain formats were preferred. Interviewees were asked to describe the format of guidelines they has used, and then asked, “Do you prefer that format over other formats? Why or why not?” With one exception, all participants preferred online access for guidelines. “Absolutely,” said one participant, “I think guidelines need to be online.” Participants were asked about print, PDF or HTML. Print was

Chapter 6 202 preferred by a single participant, but even they admitted to using an online version, “I’m sure that’s my age. Younger people would happily toggle back and forth. But a combination was fine.

I never bought the web style guide, I just used it online.” Table 6.4 shows comments regarding three formats for guidelines.

Table 6.4 Comments regarding guideline formats

Format Attitude Sample Comment Print Pro  Books are easier. You can have them in front of you.  I prefer paper because I find that sketching and drawing and sitting away from the computer helps me to work more efficiently when I sit down at the computer. So I do prefer printed guides. Con  I try to avoid printed stuff for web, because the landscape changes constantly. Print material could be old news within a month  A book format doesn’t help me if I’m in a hurry. I need something that helps me narrow down the options.  Printed versions get lost easily. PDF Pro  I would start with PDF then perhaps print them, but then that causes problems because of the difference between colour on screen and in print. Con  I find it a bit cumbersome. I appreciate the accessibility of it, the immediacy of it as opposed to a book which I may not have on hand. But I find it’s a lot easier to find and navigate through a book itself, or online resources that are presented in HTML format.  PDF for large volumes of data is harder to navigate than a book in that it’s more cumbersome, it doesn’t have the speed of hypertext, or the speed of being able to flip to a page in a book quickly. HTML Pro  Online you can either find the specs, or links to the review process that it went through so you can see the different iterations of that spec as it was being done and the review process where people submit comments at certain deadline points  I would prefer an HTML resource which people can quickly move through; find what they need in the index, click and move.  Online is what I prefer. It’s more easily parsable. It’s easier to parse and easier to get at.  The online versions are much more useful if they’re better organized  Online is certainly more convenient and certainly more reliable Con None expressed

Chapter 6 203 One participant expressed no preference, largely because their knowledge of a specific guideline came from conversations instead of directly accessing a guideline. “To be fair,” they said, after thinking about the different aspects of HTML and print, “a lot of these guidelines come to me from conversations with others regarding guidelines and they don’t always provide me with a specific reference. But I don’t have a preference. I don’t really care. I just imagine that everything is moving more and more online, and I support that.”

6.4 Summary

The interviews either supported findings from the survey (such as the nature of design trade-offs, and the use of evidence-based guidelines and guidelines supported by expert opinion) or led to a better understanding of guideline use (such as where guidelines were consulted in the design process), alternatives to guidelines (such as pattern libraries), and factors in choosing guideline sets. In addition, the interviews explored other areas not addressed in the survey, such as knowledge sharing among designers and guideline format preferences..

Based on the interviews with practitioners, guideline use involves the making of numerous decisions, all of which potentially impact their value as tools to assist web interface design. Various factors have to be addressed before a guideline is even considered, including the recency of the guideline, the trustworthiness of both the guideline recommender and the guideline itself, and whether the guideline complements the expertise of the user. This last factor illustrates the paramount importance of a designer’s experience. Guidelines are arguably not a tool for those without web interface design experience, but rather complement the existing skills of practitioners. Guidelines often codify knowledge that is already possessed by the practitioner

Chapter 6 204 and potential guideline user. This suggests a desire for guidelines to function similar to checklists or memory aids, a finding consistent with Study #1 and Study #3.

The interviews showed that the definition of what constitutes a guideline in this

dissertation differs from the definition forwarded by practitioners. The interviews with

practitioners reveal that guidelines are not limited to the model first proposed by Smith and

Mosier in 1986 (a template which informs evidence-based guideline sets and many guideline sets

supported by expert opinion), but include such relatively minimalist forms of codified

knowledge as a checklist, and expansive and organic forms such as pattern libraries. Checklists,

by definition, lack the detail of a guideline, while pattern libraries lack the structure. In the

language of the participants, guidelines are more accurately defined as codified knowledge which guides design.

Interviews revealed practitioners welcome more guidance and that they appreciate access to advice. The idea of an all encompassing set of guidelines was not even considered by participants who described guideline sets in terms suggesting a buffet. A guideline from one set was combined with a guideline from another, both of which were informed by yet a third guideline. Each guideline was judged on its own merits with the ultimate goal of resolving design problems and moving the design process forward. The notion of a set was not important.

The use of guidelines at various points in the design process was expected given the findings of Study #1 and Study #3, but the interviews also revealed how practitioners view both the role of guidelines in the process, and the role of expertise. Trade-offs faced by designers are generally not those narrowly defined in the literature, with the notable exception of the trade-off between web clarity and dense content. In Study #3 (see Table 5.5), almost two-thirds of survey respondents indicated experience with that particular trade-off. Even though the literature (and

Chapter 6 205 Study #3) considered four design focused trade-offs, web clarity versus dense content was the

only design focused trade-off discussed by the participants. The narrowness of a working definition can also be found in what the participants believed constituted trade-offs. Definitions

were not limited to specific web interface design problems, but included negotiations with clients

and non-experts involved in the design process, as well as trade-offs found in project

management literature; broad trade-offs which were not unique to web interface design. This

should not be surprising as web interface design is situated within a broader service industry, one

where client demands and project management are an important factor in successfully

completing a design and influence the design choices made to address specific and potentially

unique problems.

Design work was considered to be solitary by over half the interviewees, although

knowledge is shared by numerous means. The most common means to share knowledge was not

via electronic means (which lend themselves to sharing knowledge in the digital environment of

web interface design), but meeting face-to-face. Over half the participants made reference to face-to-face meetings, the sharing of sketchbooks, and regretted the lack of opportunities to meet and talk. If guideline sets are used, there was a stated preference for HTML formats, which offered greater ease of use for practitioners than books or even PDFs, which can be considered hybridizations of old and new technology. The findings from the interviews are similar to those from previous studies presented in this dissertation; in particular the attitudes of design students

when asked to use guidelines in a design project (see Study #1). There are echoes and threads

which run throughout the findings of the four studies, which will be discussed and explored in

the next chapter.

Chapter 6 206 Chapter 7

Research summary and discussion

Introduction

The studies described in the previous four chapters were designed in an iterative manner, where

the conclusion of one study led to the development of the next. Research questions were

addressed by multiple studies, and this chapter will begin by providing a summary of the

findings. The chapter concludes with a synthesis of the research through a discussion of the implications.

Research summary

This research was motivated by the desire to better understand how research evidence can be

shared with communities of practice and the belief that research published in academic contexts

(such as journals and conference proceedings) would benefit practitioners of web interface

design. Knowledge is shared in numerous ways, and this research specifically identified guideline sets as tools of knowledge translation. Guidelines are a means by which research, in some cases spanning decades, can be presented in a format potentially useful to practitioners.

A preliminary study with engineering students in a design course revealed attitudes

towards the use of guidelines and issues regarding guidelines that were observed in subsequent

studies; (i) the important role of personal experience as a factor in the decision to apply a

guideline to a design problem, (ii) inconsistency in the interpretation of guidelines, and (iii)

disagreements on the role of guidelines in the design process. The use of guidelines to help solve design problems was predicated by the pre-existing knowledge of the study participants, who

Chapter 7 207 suggested that they would be more apt to consider guidelines perceived as “common sense.” But when guidelines were used, the interpretations varied greatly. Participants within the same workgroup disagreed on whether a specific guideline had been applied or not, and focus group discussions revealed differing interpretation for a number of guidelines. The discussions also revealed the use of guidelines at various points in a design project (prior to design, at some point in the middle, or upon conclusion). While referencing guidelines only to support design solutions following the conclusion of the design was unexpected, the problems involving guideline interpretation were the most striking results of the study.

The second study, involving the assessment of ten health information sites by eight professional designers, revealed the interpretation of guidelines to be a critical problem.

Guideline interpretation varied between professional designers (who constituted the participant group). The interpretation of a guideline by a participant was malleable. When participants were paired, it was found that one third of their interpretations differed. Yet when asked to discuss and potentially resolve their differences, the differences were all resolved. In other words, participants revised their original interpretations, revealing the remarkably subjective nature of interpretation. Further, the resolution of differences did not result in identical interpretation across the four pairs. Consistent interpretation of all 67 guidelines used in the study was not possible.

The study also considered differences in guideline characteristics, but found no difference between guidelines on which there was greater agreement during the assessment of websites from guidelines with lower agreement rates. Advice on guideline characteristics could not be drawn from the study data. Since the study focused on only eight designers and provided them with a guideline set to use, it was unclear whether a broader group of designers recognized that

Chapter 7 208

guideline interpretation was problematic, and more importantly, if they even consulted guideline

sets in the course of their practice. A subsequent study surveyed the broad web design

community regarding their attitudes towards guidelines.

The web-based survey employed in the third study revealed that the design community defines guidelines in broad terms and the presence of research evidence was not a factor in the adoption of a guideline. Instead, guideline use depended on context of use, the ability of the designer to interpret and adapt the guideline recommendations, and the quality of the guidelines themselves. Respondents used the term “quality” in reference to the solutions or advice offered, rather than the supporting evidence for the advice. With some exceptions, research evidence was not reported to be a factor in the adoption of a guideline set.

The issue of design trade-offs was raised in the study. Respondents’ attitudes towards

trade-offs suggested a continuum that ran between strong dislike and grudging acceptance. This

acceptance spoke to the belief that the very definition of design was the selection of mutually

exclusive solutions. In other words, design was the process of addressing trade-offs. Trade-offs

specifically mentioned by respondents were not limited to choices between different design

solutions, but included trade-offs involving project stakeholders, budget and time constraints and

the limitations of technology.

The final study, involving interviews with 20 participants from the web-based survey, allowed for a focused consideration of guideline perception and use by designers. The choice of adopting a guideline depended on how many years had passed since the guideline was written, the trustworthiness of both the guideline recommender and the guideline itself, and the fit between guideline advice and the designer’s experiences. Similar to the students in the first study, the designers who participated in interviews identified their own experience as a factor in

Chapter 7 209 both the decision to apply a guideline to a design problem, and to where in the process a guideline might be consulted. Rather than tools to aid those lacking design experience, guidelines were considered a means to complement the existing skills of practitioners.

Designer’s expressed a desire for guidelines to function similar to checklists or memory aids.

The suggestion that checklists (instead of guideline sets) might better benefit the design process was an unexpected outcome which points towards future research.

Discussion

Web design guidelines are a welcome tool for designers facing design problems, but their use is problematic for a key reason; guideline interpretation is inconsistent. This supports the findings of de Sousa and Bevan (1990) and Tetzlaff and Schwartz (1991) that designers only partially understood guidelines. This does not imply a fault on the part of designers, but rather a problem with guidelines. The research described in Chapter four shows that even a pair of designers considering identical websites interpreted guidelines differently in roughly one-third of instances. While subsequent discussions between paired individual participants resolved these conflicts, the interpretation between the four pairs still differed at a comparable rate. The four pairs also disagreed on one-third of their interpretations. This suggests that the interpretation of guidelines by groups of designers (or teams) does not resolve problems found in individual interpretations. If the guidelines cannot be consistently interpreted, they cannot represent a viable means to translate research evidence. As a means to present and share research knowledge, guidelines appear to fail.

Guidelines which were interpreted consistently by participants did not differ in terms of their characteristics from guidelines with inconsistent interpretation. There is no characteristic or

Chapter 7 210 property differentiating these guidelines from each other. Cluster analysis of the assessments revealed structural relationships between guidelines, yet confirmed no relationship between differences in clusters and differences in guideline agreement rates. While the research showed that interpretation can hinge on a single word or a problematic example, no characteristic or pattern could be revealed regarding the nature of that word or example. As a result, it is not possible to provide suggestions on how to better present guidelines to eliminate, or at least reduce, inconsistent interpretation. However, the research has shown that ambiguity can result from relatively minor characteristics of a guideline and if attention is focused on words and examples used to convey the message of a guideline, there may be improvements in interpretation.

In addition, study participants offered ways that guidelines were helpful, albeit in a broader context. Guidelines provided a framework for design and could be used to impose limitations on potential design solutions. Guidelines were also helpful in supporting design choices, by reinforcing an approach to a design problem. Finally, they were thought to be helpful as memory aids or checklists. Participants who found guidelines less helpful did not differ in their opinion of guidelines. By stating that guidelines were redundant and too narrow in focus, they reveal a notion of guidelines consistent with those who favoured their use. Redundancy emerges from the expertise of the designer; where some participants found the guidelines supportive in reinforcing expertise; others found such reinforcement unnecessary and redundant.

Similarly, the perceived narrow focus of guidelines allows, on the one hand, for a limiting of design solutions, whereas on the other hand such limitations overly restrict the options available to designers. Since the perception of guidelines as either helpful or hindering is a question of

Chapter 7 211

personal preference, recommendations to improve guideline interpretation cannot be drawn from

such broad perceptions.

The reason access to guideline sets was welcomed, in spite of their shortcomings, is that

designers considered guideline sets to be an opinion to consider when facing design problems.

Designers working on sites for which accessibility was a requirement indicated that they were

prone to consult the WCAG guideline sets, since the set spoke directly to a specific design issue;

accessibility. The WCAG guideline sets were seen as key resources when accessibility was a

design requirement, even though the guidelines lacked supporting research evidence. Guidelines

serve as reminders of best practice, or as another voice in the discussion of how best to resolve a

design issue. Further, the fact that some guideline sets were supported by research evidence was

not a critical factor in their adoption. Research-based Web Design & Usability Guidelines (U.S.

Department of Health and Human Services, 2006) was used by only 11% of survey respondents

and research evidence played a minimal role in the decision to adopt this particular guideline set.

In other words, designers do not seek guidelines in order to access research evidence, but rather as a source of ideas, as they might solicit the opinions of peers, examples from other sites, books

or pattern libraries.

While some designers reported a preference regarding when guidelines are best

consulted, the complexity of the design process means that the introduction of design advice was

viable at all stages of design. When guidelines are used had some bearing on the primary trade-

off identified in the research; between the vision of the designer and the client. Design specific

trade-offs found in the literature were also identified by participants in Study #3, but when asked

about trade-offs, the interviewees in Study #4 discussed the issue of having to address client

needs, as well as the need to work within the limitations of technology, over trade-offs that were

Chapter 7 212

specifically design focused. Guidelines were viewed as a potential tool for resolving trade-offs,

not because they address specific trade-offs, but because they offer a different perspective on

both trade-offs directly related to design choices, or trade-offs involving clients or technical limitations. Guidelines were valued for offering alternative perspectives or solutions that might not have otherwise been considered. It is important to keep in mind that value subscribed to guidelines is not related to the presence of supporting research. Or rather, research evidence was not found to be a key factor in consulting a specific guideline.

Guidelines appear to be a problematic means to codify research knowledge. Research evidence is not favoured by designers over the experience of the design community. The

accumulated experience of solving design problems is a greater factor in the consideration of a

guideline than the fact that a guideline is supported by research evidence.

The knowledge-to-action framework proposed by Graham et al. (2006), with knowledge

creation surrounded by an iterative seven phase action cycle, offers a means to understand the

transfer of design research into guidelines or similar tools of knowledge transfer. In particular, the action cycle outlined by the framework mimics some of the actions reported by practitioners, who discussed the importance of selecting the appropriate knowledge for a design problem, adapting the solution to a particular context, evaluating the outcome and finally translating the knowledge to expert opinion.

While making a number of claims regarding the shortcomings of guidelines, the research in this dissertation has made a number of contributions and has identified areas of future research. The contributions will be discussed in Chapter eight, followed by limitations and future research.

Chapter 7 213

Chapter 8

Contributions, limitations and future work

8.1 Introduction

This chapter will consider the research questions which informed the studies, outline contributions, discuss limitations, and propose directions for future research.

8.2 Research answers

The six research questions are listed below, followed by answers derived from the four studies.

RQ1: What are web interface designers’ attitudes towards the use of design guidelines?

Web interface designers participating in Study #3 and #4 reported that they welcome

design guidelines, viewing them largely as aids to memory in support of existing design

experience. The use of guidelines was predicated on the existing knowledge of the

designer, the context of use, the ability to interpret and adopt the guideline

recommendation, and the perceived quality of the guideline. Where guideline

recommendations ran counter to design practices, designers expressed displeasure and

mistrust. Finally, it was reported in Study #1, #3 and #4 that guidelines were used to

justify pre-existing design solutions when clients challenge practitioners.

Chapter 8 214

RQ2: What is the level of awareness and usage of evidence-based web design guidelines by

web interface designers?

While the level of awareness and use of evidence-based web design guidelines is below

50%, a guideline being “evidence-based” is not an important factor in its potential usage.

Study #2 found the use of evidence-based guideline sets such as ISO 9241 or Research-

based Web Design & Usability Guidelines (U.S Department of Health and Human

Services, 2006) was 4% and 11% respectively.

RQ3: How are evidence-based web design guidelines interpreted by web interface designers?

Study #1 and Study #2 found that guideline interpretation is problematic, such that a

consistent interpretation of a guideline subset with the highest research evidence is

unlikely. When comparing designers, roughly one third of their guideline interpretations

were found to differ. The characteristics of a guideline are not a factor in the consistency

of interpretation and important variations in interpretation of a guideline may result from

a flawed example or a single word.

RQ4: When are guidelines used in the design process to solve design problems?

Where guidelines are used in the design process varies by designer, including those that

only use them at the start, middle or conclusion of a project, and designers who use them

at all stages. A minority of designers only use guidelines after project delivery,

supporting design solutions after they have been implemented. These findings are

supported by Study #1, Study #3 and Study #4.

Chapter 8 215

RQ5: What specific design trade-offs influence the work of designers?

The design trade-off between ensuring clarity on a page and the demands of rich content

was found in Study #3 to be the most commonly cited design trade-off, but Study #4

revealed that the most influential trade-offs do not involve design choices, but rather the

impact of client demands and technology limitations on solutions preferred by the

designer.

RQ6: How is knowledge and information shared amongst web interface designers?

Study #3 and Study #4 found that web designers share knowledge by informal means,

both electronic and face-to-face, through a small social network. Conferences or journal

publications were not favoured for knowledge sharing.

8.3 Contributions

The process of addressing research questions resulted in ten key contributions, listed below.

Empirical contributions

1. The consistent interpretation of design guidelines by interface designers is unlikely.

2. The role of guidelines in the design process is clarified; particularly their use as memory aids to support existing design experience.

3. Guidelines supported by research evidence are not preferred by designers over guidelines that were not supported by research evidence.

Chapter 8 216

4. The level of awareness and use by designers of evidence-based guideline sets has been established. Awareness of ISO 9241 or Research-based Web Design & Usability Guidelines

(U.S. Department of Health and Human Services, 2006) was less than 50% and use was 4% and

11% respectively.

5. Resolving trade-offs was found to be a fundamental task in the design process. These trade- offs include the choice of multiple valid design solutions, as well as higher level trade-offs between client demands or technology limitations and designer preferences. Regardless of the nature of the trade-offs, guidelines (including both expert opinion or evidence-based) are viewed as potential contributors to finding resolution.

Methodological contributions

6. Proposed the use of cluster analysis to find natural groupings of guidelines based on characteristics. While the results were inconclusive in this research, the method can be more broadly applied in order to better understand properties that might improve consistent guideline interpretation.

7. Described a method to study how information is interpreted by pairing evaluators after completing individual assessments. Asking paired evaluators to discuss their interpretations, in an attempt to reach consensus, provided insights into how they assigned meaning to individual guidelines.

Chapter 8 217

Theoretical contributions

8. Found the process of translating research knowledge to action in the form of guidelines is

similar in web interface design and clinical medicine. Knowledge transfer theories and

frameworks found in the field of health care offer a key to better understanding how design

research can be distributed to, and adopted by, communities of practice. Specifically, the

iterative process of identifying a design problem, finding an appropriate solution and codifying

the knowledge mimics the knowledge-to-action framework proposed by Graham et al. (2006).

The potential use of the framework is significant for guideline authors and future researchers.

9. Guidelines were found to be broadly defined by practitioners, and include standards,

conventions, rules, checklists and pattern libraries.

10. The referencing of checklists by designers is suggestive of checklists used in health care

(Gawande, 2010). The development of checklists, either in concert with guidelines as suggested by Perlman (1989) or as a means to replace guidelines, may be an important means to translate research knowledge for the design community.

8.4 Limitations

Research contributions must be considered in the context of limitations to generalization. The research questions posed were broad in scope in order to better understand a large community of practice. The studies were limited by time, methodology and participants.

Chapter 8 218

Changes to technology and user behaviour over time

The research studies span a period of three years (2007 to 2010). Both the technologies which support web interface design and the known behaviours of users are in constant flux. New versions of operating systems and browsers are introduced annually, and users of web browsers adapt over time to design solutions which may have once been problematic. This negatively impacts guideline relevance. A guideline which addresses the download speed of web content via a dial-up modem is invalidated by the widespread use of broadband connections. Similarly, the use of frames in web page design, where once relatively common, is now very rare and guidelines addressing frames are thus of limited relevance.

Shortcomings of methodology choices

It should be recognized that even though the eight participants in Study #2 were given as much time as necessary to complete 670 individual assessments, fatigue is a potential factor in how the guidelines were interpreted and the sites assessed. The issue of fatigue was also a potential factor in the meetings between pairs; where 216 to 234 assessments were reviewed and discussed prior to reaching consensus (one of these meetings was split into two sessions totaling seven hours).

The repetitive nature of the discussions (a single guideline may have been interpreted differently across all ten web sites) meant that fatigue may have been a factor in how the guidelines were interpreted.

Chapter five described the use a web based survey. Recruitment for the study could not employ random sampling, since the sample frame was unknown. As a result, it is impossible to extrapolate the survey results beyond the 116 respondents.

Chapter 8 219

Choice of participants

While three of the four studies involved the participation of professional web interface designers, claims cannot be made that these designers represent the larger design community. Although the lively debate regarding guidelines between participants suggests otherwise, the process of reaching consensus between the individuals may have been positively influenced by the presence of the researcher.

As discussed in Chapter five, the respondents to the survey in Study #3 may have been predisposed to view guideline use in a positive light. The text in the recruitment posting described the study as looking, “at the role of design guidelines and design trade-offs in the process of web interface design,” which implies a required familiarity with design guidelines.

While the study nonetheless captured data from 20 respondents who indicated not using guidelines in their work, there may certainly have been other designers who did not use guidelines and therefore declined to participate.

In terms of the challenge of external validity for the studies, this research does not maintain its findings to be replicable in all situations. Instead, it puts forth the findings within contexts limited by specific participants. The findings of this research require extension into broader communities of practice before the findings can be considered generalizable. This is further discussed in section 8.5, which suggests future research.

Chapter 8 220

8.5 Future Work

The findings from this research suggest means to better understand the role of guidelines in the design process. All four studies were exploratory in nature and together have provided insight into the attitudes of web interface designers towards web design guidelines. Building on this work, future research can be conducted in myriad directions, but are perhaps best focused on three general areas; methodology, participants and knowledge sharing.

Research Methodology

Asking eight professional web interface designers to assess 10 web sites using a subset of guidelines revealed problems with interpretation, but there are other methods to address that research question. As well, it is uncertain if evaluating sites not focused on health information would yield different results.

Study #2 allowed participants only three choices (the website adhered, did not adhere or the guideline was not applicable). Asking participants to articulate why they made a particular choice would shed further light on how the guideline was understood. Why a specific assessment was made emerged during the paired discussions, but only where there was disagreement.

Further discussion regarding the rationale for assessments would reveal more detail about how the guidelines were interpreted.

The interviews in Study #4 provide details regarding guideline use within organizations or within larger projects. A case study of a design firm or of individuals working on a design project would provide additional knowledge about how and when guidelines are consulted, their role as tools for sharing knowledge, and what might differentiate guideline sets in the broader context of a project. Such a study may allow for a better understanding of the interplay between guidelines and designer expertise. The current research shows that the choice of guideline sets,

Chapter 8 221 where in the design process they are used, and how they are used are all predicated largely on the expertise of the designer, but the relationship between knowledge internal to the designer (expert opinion) and external knowledge is unclear.

Different population groups and additional participants

The development of a sampling frame would potentially allow for generalizability. A number of organizations assisted in the recruitment for Study #3, but only as a means to contact members.

Future work could involve close cooperation with an organization such that a random sample is drawn from its members.

Studies with additional designers would assist in obtaining results that, if not generalizable, would at least benefit from a larger pool of professional expertise.

Different tools for knowledge sharing

The research reveals that designers favour access to more guidelines, in the belief that the more resources at their disposal, the better the quality of their solution to particular design problems.

Future research could better look at guideline characteristics and formats, crowd sourcing and checklists.

Guideline characteristics and formats

The characteristics of guidelines were explored following the collection of data in Study #2.

However, participants were not asked specific questions about the guideline formats nor were they exposed to variations of guideline formats. Asking participants to use guidelines with varying characteristics (for example increasing the number of examples illustrating a certain

Chapter 8 222 guideline, or including expert opinion) would help to better understand why some guidelines were interpreted in a consistent manner while other guidelines were not.

Interviewees in Study #4 expressed different preferences for the format of guidelines, describing the positive and negative aspects of using guidelines in printed or electronic formats.

The format of guidelines should be further studied in terms of its effect on interpretation and use.

Crowd Sourcing

The use of pattern libraries by study participants was described in both Study #3 and Study #4.

While some pattern libraries represent the work of a single entity, others represent the cumulative efforts of a community. The medium in which web interface designers work also allows for the sharing of knowledge and dissemination of experience through the use of wikis, message boards, and similar applications for collaboration. Pattern libraries are knowledge repositories which will potentially continue to evolve.

As discussed earlier (see Section 2.5.2) Henninger, Haynes, and Reith (1995) and

Henninger, Lu, and Faith (1997) proposed a tool which links to specific guidelines, with the long term goal to create a repository of examples to highlight which guidelines work best in a given context. Pattern libraries serve a similar function. Such libraries propose a problem, and then list numerous solutions; solutions which could grow from communities of practice. A study of pattern libraries and how they are (or might be) used by practitioners would allow us to better understand this particular form of knowledge sharing and the role it might play in the dissemination of knowledge.

Chapter 8 223

Checklists

If pattern libraries reflect an ever expanding collection of solutions to design problems,

checklists represent a narrowing of codified knowledge. Participants in Studies #1, #3, and #4

suggested the value of either memory aids or specifically referenced the need for checklists.

Unlike guidelines, checklists are an even more condensed form of information. Yet what might be lost in breadth may be gained in ease of use and relevance. Checklists, as described in the work of Gawande (2010), are not just transmitters of institutional or research knowledge, but are a means to encourage communication between team members, ensuring that key issues are discussed. The development of web interfaces is increasingly complex and as design work continues to involve numerous individuals, checklists aid in reducing complex operations to a manageable number of key components.

The development of checklists to aid in web interface design might be a useful application of research findings. Nested checklists would allow for larger volumes of knowledge to be reduced and presented in a format that might benefit designers. Checklists may well represent the next chapter in the codifying of both research knowledge and expert opinion.

8.6 Conclusion

The challenge of understanding how research evidence can be shared with web interface designers, and the role that design guidelines play in the exchange of knowledge, was the starting point for this research. As web interface designers, we struggle to find ways to incorporate research findings into the design problems we face, and appreciate the work of guideline authors in attempting to codify decades of research from various disciplines while addressing the importance of expert opinion. While some variation in how a guideline would be interpreted was

Chapter 8 224 expected, the extent of the problem was surprising. The research showed that too many guidelines were interpreted inconsistently.

These problems emerged in the first study, where it was believed that perhaps the students who took part did not have the experience to interpret guidelines. But the issues raised by this group were echoed in the discussions in the subsequent study where the participants were not students, but professional web interface designers with multiple years of experience. But guidelines were not perceived by designers as being deeply flawed. When the broader design community was asked about their use of guidelines, the survey revealed that guidelines were viewed positively and used in the course of design work. The final study resolved what at first appeared contradictory; the flaws inherent in guidelines (and guideline interpretation) did not mean they were not useful in the eyes of practitioners. Rather, guidelines help to forward the discussion, to ensure that design issues are raised, and to represent another voice at the figurative table. It is a voice welcomed by designers.

For web interface designers, research evidence might not be as important as experiential knowledge, but such evidence has a role to play in addressing design problems and addressing the trade-offs that help define the task of design. The challenge facing us now is how to better share this knowledge and ensure that the findings of research are disseminated to the broader design community in a manner that addresses their needs.

Chapter 8 225

References

Alexander, C. (1977). A Pattern Language. New York: Oxford University Press.

Apple. (2011). Mac OS X Human Interface Guidelines. Retrieved September 1, 2011 from http://developer.apple.com/library/mac/documentation/UserExperience/Conceptual/ AppleHIGuidelines/OSXHIGuidelines.pdf

amazon.ca. (2011). Browse Subjects. Retrieved from http://www.amazon.ca

Barber, K. (Ed.). (1998). The Canadian Oxford Dictionary. Toronto: Oxford University Press.

Beier, B., & Vaughan, M.W. (2003). The Bull's Eye: A Framework for Web Application User Interface Design Guidelines. Proceedings of the SIGCHI conference on Human factors in computing systems (CHI ‘03), Ft. Lauderdale, Florida.

Campbell, J. L. (1996). The development of human factors design guidelines. International Journal of Industrial Ergonomics, 18, 363-371.

Carvalho, L., & Dong, A. (2009). Legitimating design: a sociology of knowledge account of the field. . 30(5) 483-501.

Cherry, J. M. (1998). Bibliographic Displays in OPACs and Web Catalogs: How Well Do They comply with Display Guidelines? Information Technology and Libraries, 17(3), 124-137.

Cherry, J. M., Muter, P., & Szigeti, S. (2006). Bibliographic Displays in Web Catalogs: Does Conformity to Design Guidelines Correlate with User Performance? Information Technology and Libraries, 25(3), 154-162.

Chipley, M., Lawson, R. S., & Holzheimer, T. (2004). Landscaping in the age of terrorism: new design guideline can protect and defend. Urban land. 63(11-12), 109-111.

Christakis, D.A. & Rivara F.P. (1998). Pediatricians’ awareness of and attitudes about four clinical practice guidelines. Pediatrics. 101(5), 825-30.

Cloninger, C. (2002). Fresh Styles for Web Designers. Indianapolis, IN: New Riders.

References 226

Cohen, A., Crow, D., Dilli, I., Gorny, P., Hoffman, H.J., Iannella, R., Ogawa, K., et al. (1994). Tools for Working with Guidelines. SIGCHI Bulletin, 27(2), 30-32.

Connell, R.S. (2008). Survey of Web Developers in Academic Libraries. Journal of Academic Librarianship. 34(2) 121-129.

Creswell, J.W. (2003) Research Design. Thousand Oaks, CA: Sage Publications.

Curran, K. & Robinson, D. (2007). An Investigation Into Web Content Accessibility Guideline Conformance for an Aging Population. International Journal on ELearning. 6(3), 333-350.

D'Angelo, J., & Twining, J. (2000). Comprehension by Clicks: D'Angelo Standards for Web Page Design, and Time, Comprehension, and Preference. Information Technology and Libraries, 19(3), 125-135.

De Angeli, A., Sutcliffe, A., & Hartmann, J. (2006). Interaction, Usability and Aesthetics: What Influences Users' Preferences? Proceedings of the 6th conference on Designing Interactive systems (DIS ’06), University Park, Pennsylvania, 271-280.

Davis, D., Evans, M., Jadad, A., Perrier, L., Rath, D., Ryan, D., et al (2003). The case for knowledge translation: shortening the journey from evidence to effect. BMJ. 327, 33-35.

Davis, D. (2006). Continuing education, guideline implementation, and the emerging transdisciplinary field of knowledge translation. Journal of Continuing Education in the Health Professions. 26(1), 5-12.

Decker, J. (1994). The validation of computer simulations for design guideline dispute resolution. Environment and behavior. 26(3), 421-443. de Souza, F., & Bevan, N. (1990). The Use of Guidelines in Menu Interface Design: Evaluation of a draft standard. Proceedings of the IFIP TC13 Third International Conference on Human-Computer Interaction (Interact 90), Cambridge, UK.

Dillman, D. A., Smith, J.D., Christian, L.M. (2009). Internet, Mail and Mixed-Mode Surveys: The Tailored Design Method. (3rd ed.). Hobokon, NJ: John Wiley & Sons, Inc.

Estabrooks, C.A., Thompson, D.S., Lovely, J.E., & Hofmeyer, A. (2006). A Guide to Knowledge Translation Theory. The Journal of Continuing Education in the Health Professions. 26, 25-36.

References 227

Ford, R. & Wiedeman, J. (2008). Guidelines for Online Success. Koln, Germnay: Taschen Books.

Galitz, W. O. (1993). User-Interface Screen Design. Boston: QED Publishing Company.

Gawande, A. (2010). The Checklist Manifesto. New York: Metropolitan Books.

General Electric (2011). Untitled Home Page. Retrieved from http://www.ge.com/ca/en/

Gido, J. & Clements, J.P. (2009). Successful project management. (4th ed.). Mason, OH. South-Western Congage Learning.

Graham, I., Logan, J., Harrison, M.B., Straus, S.E., Tetroe, J., Caswell, W., et al. (2006). Lost in Knowledge Translation: Time for a Map? The Journal of Continuing Education in the Health Professions, 26, 13–24.

Grey, C.F. & Larson, W. (2006). Project Management: The Managerial Process. (3rd ed.). Boston: McGraw-Hill.

Grol, R., Dalhuijsen, J., Siep, T., Veld, C., Rutten, G., & Mokkink, H. (1998). Attributes of clinical guidelines that influence use of guidelines in general practice: observational study. British Medical Journal. 317, 858-861.

Hackett, S., Parmanto, B., & Zeng, X. (2005). A retrospective look at website accessibility over time. Behaviour & Information technology. 24(6), 407-417.

Havelock, R.G., Guskin, A., Frohman, M., Havelock, M., Hill, M., & Huber, J. (1969). Planning for Innovation through Dissemination and Utilization of Knowledge. Center for Research on Utilization of Scientific Knowledge: Ann Arbor, MI.

Head, M., & Hassanein, K. (2002). Web Site Usability for eBusiness Success: Dimensions, Guidelines and Evaluation Methods. Proceedings of the Hawaii International Conference on Business, Honolulu, Hawaii.

Henninger, S. (2000). A Methodology and Tools for Applying Context-Specific Usability Guidelines to Interface Design. Interacting With Computers,12(3), 225-243.

Henninger, S., Haynes, K., & Reith, M., W. (1995). A Framework for Developing Experience- Based Usability Guidelines. Proceedings of the Symposium on Designing Interactive Systems, Ann Arbor, MI.

References 228

Henninger, S., Lu, C., & Faith, C. (1997). Using Organizational Learning Techniques to Develop Context-Specific Usability Guidelines. Proceedings of the Symposium on Designing Interactive Systems, Amsterdam.

Herbruck, D. & Umbach, S. (1997). and : linking people and process. Design Management Journal. 8(2), 44-50.

Hewlett Packard. (2011). Information for Web page developers. Retrieved from http://www.hp.com/hpinfo/abouthp/accessibility/webaccessibility/web_developers.html

Hider, P., Burford, S., & Ferguson, S. (2009). The use of information architecture guidelines by Australian libraries. Journal of Web Librarianship, 3(1), 55-70.

Hunkleler, D., & Vanakair, E. (2000). EcoDesign and LCA: Survey of Current Uses of Environmental Attributes in Produce and Process Development. Ecometrics: International Journal of LCA, 5(3), 145-151.

Indigo.ca (2011). Books: Computers. Retrieved from http://www.indigo.ca/

IBM. (2011). About IBM. Retrieved from http://www.ibm.com/ibm/us/en

Iannella, R. (1995). HyperSAM: A Management Tool for Large User Interface Guideline Sets. SIGCHI Bulletin, 27(2), 42-45.

InfoDesign. (2011). InfoDesign: Understanding by design. Retrieved from http://informationdesign.org/

ISO. (2004). International Organization for Standardization Retrieved from http://www.iso.org/iso/en/aboutiso/introduction/index.html

ISO. (1998). ISO 9241-11: Ergonomic requirements for office work with visual display terminals (VDTs) – Part 11: Guidance on usability. International Organization for Standardization. Geneva, Switzerland.

ISO. (1997a). ISO 9241-12: Ergonomic requirements for office work with visual display terminals (VDTs) – Part 12: Presentation of information. International Organization for Standardization. Geneva, Switzerland.

References 229

ISO. (1997b). ISO 9241-14: Ergonomic requirements for office work with visual display terminals (VDTs) – Part 14: Menu Dialogues. International Organization for Standardization. Geneva, Switzerland.

Ivory, M., & Megraw, R. (2005). Evolution of Web Site Design Patterns. ACM Transactions on Information Systems, 23(4), 463-497.

Ivory, M., Sinha, R., & Hearst, M. (2001). Empirically Validated Web Page Design Metrics. Proceedings of SIGCHI 01. Seattle, WA.

The Interaction Design Association. (2011). Welcome to the Interaction Design Association. Retrieved from http://www.ixda.org/

Janeiro, J., Barbosa, S.D.J., Springer, T. & Schill, A. (2010). Semantically relating user interface design patterns. Proceedings of Pattern-drive engineering of interactive computing systems (PEICS) at the ACM SIGCHI Symposium on Engineering Interactive Computing Systems. Berlin, Germany.

Janeiro, J., Barbosa, S.D.J., Springer, T. & Schill, A. (2009). Enhancing User Interface Design Patterns with Structures. Proceedings of the Special Interest Group on Design of Communication (SIGDOC 09). Bloomington, Indiana.

Jokela, T., Ilavari, N., Matero, J., & Karukka, M. (2003). The Standard of User-Centred Design and the Standard Definition of Usability: Analyzing ISO 13407 against ISO 9241-11. Paper presented at the Proceedings of the Latin American conference on Human computer interaction. Rio de Janeiro, Brazil.

Kabir, I. (2008). Representation and Reorganization of Web Accessibility Guidelines Using Goal Graphs and Design Patterns. Unpublished master’s thesis. University of Toronto, Toronto.

Karam, E., O’Connell, T., & Perry, J. (2009). Information Resources Utilized by the Engineering Design Community. Worcester Polytechnic Institute. BSc thesis.

Kim, H. (2010.) Effective organization of design guidelines reflecting designer's design strategies. International Journal of Industrial Ergonomics. 40(6), 669-688.

Kim, J., Lee, J., & Choi, D. (2003). Designing emotionally evocative homepages: an empirical study of the quantitative relations between design factors and emotional dimensions. International Journal of Human-Computer Interaction, 59, 899-940.

References 230

Koukouletsos, K., Khazaei, B., Dearden, A. & Ozcan, M. (2009). Creativity and HCI: From Experience to Design in Education. Proceedings of the IFIP International Federation for Information Processing. Springer: Boston. 289, 159–174.

Koyani, S. J., Bailey, R.W., & Nall, J.R. (2003). Research-Based Web Design & Usability Guidelines. Retrieved September 26, 2004, from http://usability.gov/pdfs/guidelines.html (publication currently unavailable)

Koyani, S., & Allison, S. (2003). Use of Research-Based Guidelines in the Development of Websites. Proceedings of the Association for Computing Machinery (CHI’03), St. Lauderdale, Florida.

Landay, J.A., & Borriello, G. (2003). Design Patterns for Ubiquitous Computing. Computer. 36(8), 93-95.

Laurel, B. (1990). The Art of Human-Computer Interface Design. Reading, MA: Addison- Wesley Publishing Company, Inc.

Linderman, M., & Fried, J. (2004). Defensive Design for the Web. Indianapolis, IN: New Riders.

Lisle, L., Dong, J., & Isensee, S. (1998). Case Study of Development of an Ease of Use Web Site. Proceedings of the 4th Conference on Human Factors & the Web. Austin, Texas. Retrieved September 1, 2011 from http://research.microsoft.com/en-us/um/people/marycz/hfweb98/lisle/

Lunt, P. & Livingstone, S. (1996). Rethinking the focus group in media and communications research. Journal of Communication, 46(2), 79-98.

McNeil, P. (2010). The Web Designer's Idea Book, Vol. 2: More of the Best Themes, Trends and Styles in Website Design. N.p.: How.

Michlewski, K. (2008). Uncovering Design Attitude: Inside the Culture of Designers. Organization Studies. 29(03), 374-392.

Microsoft. (2011). Design Guidelines for Class Library Developers. Retrieved September 1, 2011 from http://msdn.microsoft.com/en-us/library/czefa0ke%28v=vs.71%29.aspx

References 231

Mosca, L., Linfante, A.H., Benjamin, E.J., Berra, K., Hayes, S.N., Walsh, B.W., et al. (2005). National study of physician awareness and adherence to cardiovascular disease prevention guidelines. Circulation. 111(4), 499-510.

National Cancer Institute. (2001). Web Design & Usability Guidelines. Retrieved September 23, 2004 from http://usability.gov/guidelines/ (publication currently unavailable)

Nielsen, J. (2011a). useit.com: Jakob Nielsen’s Website. Retrieved from http://www.useit.com

Nielsen, J. (2011b). Current Issues in Web Usability (1995-2011). Retrieved from http://www.useit.com/alertbox/

Nielsen, J. (2010). Usability of Websites for Children: Design Guidelines for Targeting Users Aged 3-12 Years. Fremont, CA.

Nielsen, J. (2002). Usability of Websites for Children: 70 Design Guidelines. Fremont, CA.

Nielsen, J. (2000). Designing Web Usability. Indianapolis, IN: New Riders.

Nielsen, J. (1994a). Enhancing the explanatory power of usability heuristics. Proceedings of the Association for Computing Machinery (CHI'94), Boston.

Nielsen, J. (1994b). Heuristic evaluation. In Nielsen, J. & Mack, R.L. (Eds.), Usability Inspection Methods, New York: John Wiley & Sons.

Nielsen, J. & Norman, D.A. (2011). NN/g. Retrieved from http://www.nngroup.com/

Nielsen, J. & Tahir, M. (2001). Homepage Usability: 50 Websites Deconstructed. Indianapolis, IN: New Riders.

Nielsen Norman Group. (2006). Email Newsletter Usability: 149 Design Guidelines for Newsletter Subscription, Content, and Account Maintenance Based on Usability Studies. Fremont, CA.

Norman, D. A. (2004). Emotional Design. New York: Basic Books.

Norman, D. A. (1988). The Psychology of Everyday Things. Toronto: Currency Doubleday.

References 232

Norman, D. A. (1983). Design Principles for Human-Computer Interfaces. Proceedings of the SIGCHI conference on Human Factors in Computing Systems (CHI’83), Boston, 1-10.

Nuckols, T.K., Lim, Y., Wynn, B.O., Mattke, S., MacLean, C.H., Harber, P., et al. (2007). Rigorous Development does not Ensure that Guidelines are Acceptable to a Panel of Knowledgeable Providers. Journal of General Internal Medicine. 23(1), 37-44.

Open Source Design Pattern Library. (2011). Retrieved from http://www.uidesignpatterns.org/

Parsons, H. M. (1970). The scope of human factors in computer-based data processing systems. Human Factors, 19, 55-61.

Pearrow, M. (2000). Web Site Usability Handbook. Rockland, MA: Charles River Media.

Perlman, G. (1989). Asynchronous Design/Evaluation Methods for Hypertext Technology Development. Proceedings of ACM Hypertext ‘89 Conference. Pittsburgh, Pennsylvania, 61-82.

Pronovost, P., Needham, D., Berenholtz, S., Sinopoli, D., Chu, H., Cosgrove, S. et al. (2006). An intervention to decrease catheter-related bloodstream infections in the ICU. New England Journal of Medicine. 355(26), 2725-2732

Ray, S. D. (2002). Web Guidelines & Usability. PhD, Indiana University.

Regan, B. (2004). Accessibility and Design: A Failure of the Imagination. Proceedings of the 2004 International cross-disciplinary workshop on Web accessibility (W4A), New York. 29-37

Rogers, E. M. (2005). Diffusion of Innovations. (5th ed.). New York: Free Press.

Rogers, M. (2010). Government Accessibility Standards and WCAG 2.0. Retrieved September 1, 2011 from http://blog.powermapper.com/blog/post/Government- Accessibility-Standards.aspx

Salter, A., & Gann, D. (2003). Sources of ideas for innovation in engineering design. Research Policy. 32(8), 1309-1324.

References 233

Seffah, A. (2010). The evolution of design patterns in HCI: From pattern languages to Pattern oriented design. Proceedings of Pattern-drive engineering of interactive computing systems (PEICS) at the ACM SIGCHI Symposium on Engineering Interactive Computing Systems. Berlin, Germany.

Serce, F.C., Alpaslan, F.N., Swigger, K., Brazile, R., Dafoulas, G., & Lopez, V. (2010). Strategies and guidelines for building effective distributed learning teams in higher education. Proceedings of the 9th International Conference on Information Technology Based Higher Education and Training (ITHET). 247-253.

Shackel, B. (1991). Usability - Context, framework, definition, design and evaluation. In B. Shackel, Richardson, S. J. (Ed.), Human factors for informatics usability Cambridge, UK: Cambridge University Press. 21-37.

Shaneyfelt, T.M., & Centor, R.M. (2009). Reassessment of Clinical Practice Guidelines. The Journal of the American Medical Association. 301(8). 868-869.

Shneiderman, B. & Plaisant, C. (2009). Designing the User Interface. (5th ed.). Boston: Addison-Wesley.

Shneiderman, B. (1987). Designing the User Interface: Strategies for Effective Human- Computer Interaction. Reading, MA: Addison-Wesley Publishing Company.

Sim, J. (1998). Collecting and analysing qualitative data: issues raised by the focus group. Journal of Advanced Nursing, 28, 345–352

Smith, S. L., & Aucella, A. F. (1983). Design guidelines for the user interface to computer-based information systems. Hanscom Airforce Base, MA: U.S.A.F. Electronic Systems Division.

Smith, S. L., & Mosier, J.N. (1986). Guidelines for Designing User Interface Software. Bedford, MA: Mitre.

Straus, S.E. (2009). Defining knowledge translation. Canadian Medical Association Journal. 181(3-4), 165-168.

Sutcliffe, A. G., Kurniawan, S., & Shin, J-E. (2005). A method and advisor tool for multimedia user interface design. International Journal of Human-Computer Interaction, 64, 375-392.

References 234

Tang, A., & Aleti, A. (2010). What makes software design effective? Design Studies. 31(6), 614-640.

Tetzlaff, L., & Schwartz, D.R. (1991). The Use of Guidelines in Interface Design. Proceedings of the Conference on Human Factors in Computing Systems CHI '91. New York.

Todd, E., Kemp, E. & Phillips, C. (2004). What makes a good user interface pattern language? Proceedings of the 5th Australian User Interface Conference. Dunedin, Australia.

Tricoci, P., Allen. J.M., Kramer, J.M., Califf, R.M., & Smith, S.C. (2009). Scientific Evidence Underlying the ACC/AHA Clinical Practice Guidelines. The Journal of the American Medical Association. 301(8), 831-841.

Tripathi, P., Pandey, M., Bharti, D. (2010) Computer and Automation Engineering (ICCAE). Proceedings of the 2nd International Conference on Computer and Automation Engineering. (2) 393-397.

Tullis, T. S. (1997). Screen Design. In M. Helander, Landauer, T. K., & Prabhu, P. (Ed.), Handbook of Human-Computer Interaction (2nd ed., pp. 505-531). Amsterdam: Elsevier.

UI Pattern Library. (2011). Retrieved from http://ui-patterns.com/

U.S. Department of Health and Human Services. (2006). Research-Based Web Design & Usability Guidelines. Retrieved November 9, 2006 from http://usability.gov/pdfs/guidelines.html

U.S. Federal Aviation Administration. (2011). Retrieved from http://www.faa.gov/

Vanderdonckt, J., Mariage, C., Scapin, D., Leulier, C., Bastien, C., Farenc, C., et al. (2000). A Framework for Organizing Web Usability Guidelines. Proceedings of the 6th Conference on Human Factors & the Web. Austin, TX.

Vanderdonckt, J. & Simarro, F.M. (2010). Generative pattern-based design of user Interfaces. Proceedings of Pattern-drive engineering of interactive computing systems (PEICS) at the ACM SIGCHI Symposium on Engineering Interactive Computing Systems. Berlin, Germany.

Veen, J. (2001). The Art & Science of Web Design. Indianapolis, IN: New Riders.

Vora, P. (2009). Web application design patterns. Boston: Morgan Kaufmann.

References 235

Weinman, L. (2002). Designing Web Graphics.4. Indianapolis, IN: New Riders.

Welie.com. (2011). Patterns in Interaction Design. Retrieved from http://www.welie.com/patterns/

White, M. (2010). Developing guidelines for good practice in participatory arts-in-health-care contexts. Journal of Applied Arts and Health. 1(2), 139-155.

WC3. (2008). Web Content Accessibility Guidelines 2.0. Retrieved from http://www.w3.org/TR/WCAG20/

WC3. (1999). Web Content Accessibility Guidelines 1.0. Retrieved from http://www.w3.org/TR/WAI-WEBCONTENT/

Yahoo! Design Pattern Library. (2011). Retrieved from http://developer.yahoo.com/ypatterns/

References 236

Appendix A: Havelock’s seven factors which aid in understanding the phenomena of dissemination and utilization (D&U)

Factors which aid in Description understanding D&U phenomena Linkage The number, variety and mutuality of a resource system Structure The degree of systematic organization and coordination of related systems and D&U strategy Openness The willingness of actors to exchange information and value change Capacity The ability to retrieve and manipulate resources such as wealth, power, education, experience and the number and diversity of existing linkages Reward Positive reinforcements inherent in the linkages Proximity Nearness in time, place and context Synergy The number, variety, frequency and persistence of forces that can be mobilized to produce knowledge utilization

Adopted from: Havelock, R.G., Guskin, A., Frohman, M., Havelock, M., Hill, M., & Huber, J. (1969). Planning for Innovation through Dissemination and Utilization of Knowledge. Center for Research on Utilization of Scientific Knowledge: Ann Arbor, MI.

Appendices 237

Appendix B: Illustration of Bull’s‐Eye Framework

From Beier, B., & Vaughan, M.W. (2003). The Bull's Eye: A Framework for Web Application UserInterface Design Guidelines. Paper presented at the CHI 2003, Ft. Lauderdale, Florida. Page 490.

Appendices 238

Appendix C: Questionnaire distributed to participants (Study #1)

Study Title: The usage of interface design guidelines

Group Number: ______

Instructions: This document includes all the guidelines from the guideline booklet distributed at the start of the study. There are two parts below each guideline:  Part A asks that you please indicate in the checkbox whether you applied the guideline, applied part of the guideline or did not apply the guideline to your wiki page design.  In Part B, please provide either detail regarding how the guideline was used, your rationale for using only part of the guideline, or your reason for not applying the guideline. ______

Guideline 6:2 Put important, clickable items in the same locations, and closer to the top of the page, where their location can be better estimated.

Part A: Check one box: Applied Partially applied Not applied to design to design to design

Part B: Description or Rationale:  If the guideline was applied, please describe how.  If the guideline was partially applied, explained why only part of the guideline was used and how the guideline was partially applied  If the guideline was not applied, please explain why.

Appendices 239

Guideline 6:3 Put the most important items at the top center of the Web page to facilitate users’ finding the information.

Part A: Check one box: Applied Partially applied Not applied to design to design to design

Part B: Description or Rationale:  If the guideline was applied, please describe how.  If the guideline was partially applied, explained why only part of the guideline was used and how the guideline was partially applied  If the guideline was not applied, please explain why.

use back of page if necessary

Guideline 6:4 Structure pages so that items can be easily compared when users must analyze those items to discern similarities, differences, trends, and relationships.

Part A: Check one box: Applied Partially applied Not applied to design to design to design

Part B: Description or Rationale:  If the guideline was applied, please describe how.  If the guideline was partially applied, explained why only part of the guideline was used and how the guideline was partially applied  If the guideline was not applied, please explain why.

use back of page if necessary

Appendices 240

Guideline 6:7 Visually align page elements, either vertically or horizontally.

Part A: Check one box: Applied Partially applied Not applied to design to design to design

Part B: Description or Rationale:  If the guideline was applied, please describe how.  If the guideline was partially applied, explained why only part of the guideline was used and how the guideline was partially applied  If the guideline was not applied, please explain why.

use back of page if necessary

Guideline 6:9 Ensure that the location of headings and other page elements does not create the illusion that users have reached the top or bottom of a page when they have not.

Part A: Check one box: Applied Partially applied Not applied to design to design to design

Part B: Description or Rationale:  If the guideline was applied, please describe how.  If the guideline was partially applied, explained why only part of the guideline was used and how the guideline was partially applied  If the guideline was not applied, please explain why.

use back of page if necessary

Appendices 241

Guideline 6:11 Limit the amount of white space (areas without text, graphics, etc.) on pages that are used for scanning and searching.

Part A: Check one box: Applied Partially applied Not applied to design to design to design

Part B: Description or Rationale:  If the guideline was applied, please describe how.  If the guideline was partially applied, explained why only part of the guideline was used and how the guideline was partially applied  If the guideline was not applied, please explain why.

use back of page if necessary

Guideline 6:12 If reading speed is most important, use longer line lengths (75-100 characters per line). If acceptance of the Web site is most important, use shorter line lengths (fifty characters per line).

Part A: Check one box: Applied Partially applied Not applied to design to design to design

Part B: Description or Rationale:  If the guideline was applied, please describe how.  If the guideline was partially applied, explained why only part of the guideline was used and how the guideline was partially applied  If the guideline was not applied, please explain why.

use back of page if necessary

Appendices 242

Guideline 6:13 Use frames when certain functions must remain visible on the screen as the user accesses other information on the site.

Part A: Check one box: Applied Partially applied Not applied to design to design to design

Part B: Description or Rationale:  If the guideline was applied, please describe how.  If the guideline was partially applied, explained why only part of the guideline was used and how the guideline was partially applied  If the guideline was not applied, please explain why.

use back of page if necessary

Guideline 11:1 When users are expected to rapidly read and understand prose text, use black text on a plain, high-contrast, non-patterned background.

Part A: Check one box: Applied Partially applied Not applied to design to design to design

Part B: Description or Rationale:  If the guideline was applied, please describe how.  If the guideline was partially applied, explained why only part of the guideline was used and how the guideline was partially applied  If the guideline was not applied, please explain why.

use back of page if necessary

Appendices 243

Guideline 11:4 Ensure visual consistency of Web site elements within and between Web pages.

Part A: Check one box: Applied Partially applied Not applied to design to design to design

Part B: Description or Rationale:  If the guideline was applied, please describe how.  If the guideline was partially applied, explained why only part of the guideline was used and how the guideline was partially applied  If the guideline was not applied, please explain why.

use back of page if necessary

Guideline 11:6 Use attention-attracting features with caution and only when they are highly relevant.

Part A: Check one box: Applied Partially applied Not applied to design to design to design

Part B: Description or Rationale:  If the guideline was applied, please describe how.  If the guideline was partially applied, explained why only part of the guideline was used and how the guideline was partially applied  If the guideline was not applied, please explain why.

use back of page if necessary

Appendices 244

Guideline 11:7 Use a familiar font to achieve the best possible reading speed.

Part A: Check one box: Applied Partially applied Not applied to design to design to design

Part B: Description or Rationale:  If the guideline was applied, please describe how.  If the guideline was partially applied, explained why only part of the guideline was used and how the guideline was partially applied  If the guideline was not applied, please explain why.

use back of page if necessary

Guideline 11:8 Use at least a 12-point font (e.g., typeface) on all Web pages.

Part A: Check one box: Applied Partially applied Not applied to design to design to design

Part B: Description or Rationale:  If the guideline was applied, please describe how.  If the guideline was partially applied, explained why only part of the guideline was used and how the guideline was partially applied  If the guideline was not applied, please explain why.

use back of page if necessary

Appendices 245

Guideline 11:9 When using color-coding on your Web site, be sure that the coding scheme can be quickly and easily understood.

Part A: Check one box: Applied Partially applied Not applied to design to design to design

Part B: Description or Rationale:  If the guideline was applied, please describe how.  If the guideline was partially applied, explained why only part of the guideline was used and how the guideline was partially applied  If the guideline was not applied, please explain why.

use back of page if necessary

Guideline 11:10 Change the font characteristics to emphasize the importance of a word or short phrase.

Part A: Check one box: Applied Partially applied Not applied to design to design to design

Part B: Description or Rationale:  If the guideline was applied, please describe how.  If the guideline was partially applied, explained why only part of the guideline was used and how the guideline was partially applied  If the guideline was not applied, please explain why.

use back of page if necessary

Appendices 246

Guideline 14:1 Use background images sparingly and make sure they are simple, especially if they are used behind text.

Part A: Check one box: Applied Partially applied Not applied to design to design to design

Part B: Description or Rationale:  If the guideline was applied, please describe how.  If the guideline was partially applied, explained why only part of the guideline was used and how the guideline was partially applied  If the guideline was not applied, please explain why.

use back of page if necessary

Guideline 14:2 Ensure that all clickable images are either labeled or readily understood by typical users.

Part A: Check one box: Applied Partially applied Not applied to design to design to design

Part B: Description or Rationale:  If the guideline was applied, please describe how.  If the guideline was partially applied, explained why only part of the guideline was used and how the guideline was partially applied  If the guideline was not applied, please explain why.

use back of page if necessary

Appendices 247

Guideline 14:3 Take steps to ensure that images on the Web site do not slow page download times unnecessarily.

Part A: Check one box: Applied Partially applied Not applied to design to design to design

Part B: Description or Rationale:  If the guideline was applied, please describe how.  If the guideline was partially applied, explained why only part of the guideline was used and how the guideline was partially applied  If the guideline was not applied, please explain why.

use back of page if necessary

Guideline 14:4 Use video, animation, and audio only when they help to convey, or are supportive of, the Web site’s message or other content.

Part A: Check one box: Applied Partially applied Not applied to design to design to design

Part B: Description or Rationale:  If the guideline was applied, please describe how.  If the guideline was partially applied, explained why only part of the guideline was used and how the guideline was partially applied  If the guideline was not applied, please explain why.

use back of page if necessary

Appendices 248

Guideline 14:5 Place your organization’s logo in a consistent place on every page.

Part A: Check one box: Applied Partially applied Not applied to design to design to design

Part B: Description or Rationale:  If the guideline was applied, please describe how.  If the guideline was partially applied, explained why only part of the guideline was used and how the guideline was partially applied  If the guideline was not applied, please explain why.

use back of page if necessary

Guideline 14:6 Do not make important images look like banner advertisements or gratuitous decorations.

Part A: Check one box: Applied Partially applied Not applied to design to design to design

Part B: Description or Rationale:  If the guideline was applied, please describe how.  If the guideline was partially applied, explained why only part of the guideline was used and how the guideline was partially applied  If the guideline was not applied, please explain why.

use back of page if necessary

Appendices 249

Guideline 14:10 Include actual data values with graphical displays of data when precise reading of the data is required.

Part A: Check one box: Applied Partially applied Not applied to design to design to design

Part B: Description or Rationale:  If the guideline was applied, please describe how.  If the guideline was partially applied, explained why only part of the guideline was used and how the guideline was partially applied  If the guideline was not applied, please explain why.

use back of page if necessary

Guideline 14:11 Use a graphic format to display data.

Part A: Check one box: Applied Partially applied Not applied to design to design to design

Part B: Description or Rationale:  If the guideline was applied, please describe how.  If the guideline was partially applied, explained why only part of the guideline was used and how the guideline was partially applied  If the guideline was not applied, please explain why.

use back of page if necessary

Appendices 250

Guideline 14:13 Use images that look like real-world items when appropriate.

Part A: Check one box: Applied Partially applied Not applied to design to design to design

Part B: Description or Rationale:  If the guideline was applied, please describe how.  If the guideline was partially applied, explained why only part of the guideline was used and how the guideline was partially applied  If the guideline was not applied, please explain why.

use back of page if necessary

Guideline 14:15 To facilitate learning, use images rather than text whenever possible.

Part A: Check one box: Applied Partially applied Not applied to design to design to design

Part B: Description or Rationale:  If the guideline was applied, please describe how.  If the guideline was partially applied, explained why only part of the guideline was used and how the guideline was partially applied  If the guideline was not applied, please explain why.

use back of page if necessary

Appendices 251

Appendix D: Focus group discussion script (Study #1)

Thank you for taking time for today’s focus group. My name is Steve Szigeti and I'm a doctoral student with the Faculty of Information Studies.

First, let me explain what a focus group is, why I’m running this focus group and what I hope to learn. A focus group is a controlled, planned discussion that gathers detailed information about a certain topic. I want to better understand your experiences working with design guidelines. I hope that this information will help me better understand how guidelines are used and to see if there are some guidelines which lend themselves better to usage than others. This will also help me understand what follow-up studies are needed.

This focus group is one of two being run concurrently. Your comments today will be audio recorded so I ask that you please speak one at a time. I want to assure you that what you say is for information gathering purposes only. Your name will not be attached to your comments. I will transcribe the audio recording within a week or two, after which the audio will be destroyed. I also ask that you protect each other's privacy and not discuss today's meeting with others. Are there any questions before we begin?

1. To begin the discussion, let's go around the table, introduce yourself, tell me your role in the design group and your initial thoughts regarding the guideline booklet. (Continue until all have participated.)

2. Again going around the table, let me know if there were any guidelines that you thought were unclear? (Follow-up: If it is unclear, what do you think would improve the guideline? How could it be clarified?)

3. Anyone can answer here, but were there guidelines that you thought were good examples of how guidelines should be presented? (Follow-up: what made those guidelines good examples? What could the authors learn from those guidelines to improve the others?)

Appendices 252

4. For those in the room who worked directly with the designs, what were your initial thoughts when you considered applying the guidelines? (Follow up: was it clear what you needed to do to apply a specific guideline?)

5. Let’s look at the specific designs (print versions of the interfaces will be shared at this point). Can the group who worked on this explain how the guidelines were applied? (The same question will be asked of the other group)

6. Were there any guidelines that you only partially used? If so, show me how you used those guidelines. (Follow up: why did you decide to use only part of the guideline?) (The same question will be asked of the other group)

7. What about the guidelines you decided not to apply. Tell me about those and why you think it was best not to apply them to your design. (Follow-up: apart from what we discussed earlier, is there a way you’d suggest improving those guidelines so that someone might use them?) (The same question will be asked of the other group)

8. To finish up, let’s go around the table again. I’m curious now that you’ve had some experience using a guideline set, whether you would use this set (or a similar set) in any future design projects.

9. Would anyone like to share additional insights they gained in using the guideline booklet?

Thank you for your participation in this focus group discussion.

Appendices 253

Appendix E: Guideline subset (Study #2)

Guideline ID Guideline Title

2:3 Standardize Task Sequences 2:5 Design For Working Memory Limitations 2:6 Minimize Page Download Time 2:10 Provide Feedback When Users Must Wait 2:13 Do Not Require Users to Multitask While Reading 3:3 Do Not Use Color Alone to Convey Information 5:3 Create a Positive First Impression of Your Site 5:6 Ensure the Homepage Looks like a Homepage 6:2 Place Important Items Consistently 6:3 Place Important Items at Top Center 6:4 Structure for Easy Comparison 6:7 Align Items on a Page 6:9 Avoid Scroll Stoppers 6:11 Use Moderate White Space 6:12 Choose Appropriate Line Lengths 6:13 Use Frames When Functions Must Remain Accessible 7:8 Keep Navigation‐only Pages Short 7:9 Use Appropriate Menu Types 7:10 Use Site Maps 8:1 Eliminate Horizontal Scrolling 8:2 Facilitate Rapid Scrolling While Reading 8:3 Use Scrolling Pages for Reading Comprehension 8:4 Use Paging Rather Than Scrolling 9:1 Use Clear Category Labels 9:3 Use Descriptive Headings Liberally 10:1 Use Meaningful Link labels 10:3 Match Link Names with Their Destination Pages 10:5 Repeat Important Links 10:6 Use Text for Links 10:9 Ensure that Embedded Links are Descriptive 11:1 Use Black Text on Plain, High‐Contrast Backgrounds 11:4 Ensure Visual Consistency 11:6 Use Attention‐Attracting Features when Appropriate 11:7 Use Familiar Fonts 11:8 Use at Least 12‐Point Font 11:9 Color‐Coding and Instructions 11:10 Emphasize Importance 12:1 Order Elements to Maximize User Performance 12:2 Place Important Items at Top of the List 12:3 Format Lists to Ease Scanning 12:4 Display Related Items in Lists 12:5 Introduce Each List 12:8 Use Appropriate List Style

Appendices 254

Guideline ID Guideline Title 13:13 Use a Single Data Entry Method 13:22 Use Data Entry Fields to Speed Performance 13:25 Minimize Use of the Shift Key 14:1 Use Simple Background Images 14:2 Label Clickable Images 14:3 Ensure that Images Do Not Slow Downloads 14:4 Use Video, Animation, and Audio Meaningfully 14:5 Include Logos 14:6 Graphics Should Not Look Like Banner Ads 14:10 Include Actual Data with Data Graphics 14:11 Display Monitoring Information Graphically 14:13 Emulate Real‐World Objects 14:15 Use Images to Facilitate Learning 15:1 Make Action Sequences Clear 15:2 Avoid Jargon 15:6 Use Mixed Case with Prose 15:7 Limit the Number of Words and Sentences 15:9 Use Active Voice 15:11 Make First Sentences Descriptive 16:1 Organize Information Clearly 16:2 Facilitate Scanning 16:4 Group Related Elements 16:7 Display Only Necessary Information 16:9 Use Color for Grouping

Appendices 255

Appendix F: Assessment checklist (Study #2)

HHS Does (2006) Guideline Not # ID Guideline Title Meet Meet Uncertain Notes 1 2:3 Standardize Task Sequences 2 2:5 Design For Working Memory Limitations 3 2:6 Minimize Page Download Time 4 2:10 Provide Feedback When Users Must Wait Do Not Require Users to Multitask While 5 2:13 Reading

Do Not Use Color Alone to Convey 6 3:3 Information Create a Positive First Impression of Your 7 5:3 Site Ensure the Homepage Looks like a 8 5:6 Homepage 9 6:2 Place Important Items Consistently 10 6:3 Place Important Items at Top Center

11 6:4 Structure for Easy Comparison 12 6:7 Align Items on a Page 13 6:9 Avoid Scroll Stoppers 14 6:11 Use Moderate White Space 15 6:12 Choose Appropriate Line Lengths

Appendices 256

HHS Does (2006) Guideline Not # ID Guideline Title Meet Meet Uncertain Notes Use Frames When Functions Must Remain 16 6:13 Accessible

17 7:8 Keep Navigation‐only Pages Short

18 7:9 Use Appropriate Menu Types

19 7:10 Use Site Maps

20 8:1 Eliminate Horizontal Scrolling

21 8:2 Facilitate Rapid Scrolling While Reading Use Scrolling Pages for Reading 22 8:3 Comprehension

23 8:4 Use Paging Rather Than Scrolling

24 9:1 Use Clear Category Labels

25 9:3 Use Descriptive Headings Liberally

26 10:1 Use Meaningful Link labels Match Link Names with Their Destination 27 10:3 Pages

28 10:5 Repeat Important Links

29 10:6 Use Text for Links

30 10:9 Ensure that Embedded Links are Descriptive

Appendices 257

HHS Does (2006) Guideline Not # ID Guideline Title Meet Meet Uncertain Notes Use Black Text on Plain, High‐Contrast 31 11:1 Backgrounds 33 11:4 Ensure Visual Consistency Use Attention‐Attracting Features when 34 11:6 Appropriate 35 11:7 Use Familiar Fonts 36 11:8 Use at Least 12‐Point Font

37 11:9 Color‐Coding and Instructions 32 11:10 Emphasize Importance Order Elements to Maximize User 38 12:1 Performance 39 12:2 Place Important Items at Top of the List 40 12:3 Format Lists to Ease Scanning

41 12:4 Display Related Items in Lists 42 12:5 Introduce Each List 43 12:8 Use Appropriate List Style Use Radio Buttons for Mutually Exclusive 47 13:9 Selections 44 13:13 Use a Single Data Entry Method

Appendices 258

HHS Does (2006) Guideline Not # ID Guideline Title Meet Meet Uncertain Notes 45 13:22 Use Data Entry Fields to Speed Performance 46 13:25 Minimize Use of the Shift Key 48 14:1 Use Simple Background Images 49 14:10 Include Actual Data with Data Graphics 50 14:11 Display Monitoring Information Graphically

51 14:13 Emulate Real‐World Objects 52 14:15 Use Images to Facilitate Learning 53 14:2 Label Clickable Images 54 14:3 Ensure that Images Do Not Slow Downloads Use Video, Animation, and Audio 55 14:4 Meaningfully

56 14:5 Include Logos 57 14:6 Graphics Should Not Look Like Banner Ads 58 15:1 Make Action Sequences Clear 60 15:2 Avoid Jargon 61 15:6 Use Mixed Case with Prose

HHS Does (2006) Guideline Not # ID Guideline Title Meet Meet Uncertain Notes

62 15:7 Limit the Number of Words and Sentences

63 15:9 Use Active Voice

59 15:11 Make First Sentences Descriptive

64 16:1 Organize Information Clearly

65 16:2 Facilitate Scanning

66 16:4 Group Related Elements

67 16:7 Display Only Necessary Information

68 16:9 Use Color for Grouping

Appendices 259

Appendix G: Questionnaire Instrument (Study #3)

The following 11 pages (261-271) have been output from www.surveymonkey.net, where the questionnaire instrument resided during the study. This print version divides each of the online pages via a grey horizontal band with the page title.

Appendices 260

Welcome

Thank you for taking time to participate in this study, conducted by Steve Szigeti, doctoral candidate in the Faculty of Information, University of Toronto.

Your responses are very important. This survey gives you an opportunity to express your opinions and thoughts about web interface design guidelines and design trade­offs.

The questionnaire takes about 15­20 minutes to complete.

All of your responses will remain anonymous and confidential. If you do not wish to respond to particular questions, please skip over them. Please know that you can refuse to answer any question, or withdraw at any time.

If you have any questions, feel free to contact me at [email protected].

Click on the NEXT PAGE button to begin the survey. In doing so, you are consenting to participate in the research as per the conditions outlined above. On the final screen, I will again ask you to confirm your consent.

Thank you

Thank you for your interest. Unfortunately our research seeks only active web interface designers as participants.

Please close your browser window to complete your participation.

Thank you

Thank you for your interest. Unfortunately our research seeks only active web interface designers as participants.

Please close your browser window to complete your participation.

Section A: Design Work

In this first section, I ask you to share your web interface design experiences.

A1. Do you currently design web interfaces?

nmlkj Yes

nmlkj No

Section A: Design Work A2. How many years have you worked as a web interface designer?

nmlkj less than 1 year

nmlkj 1­3 years

nmlkj 4­6 years

nmlkj 7­10 years

nmlkj 11­15 years

nmlkj 16­20 years

nmlkj 21­25 years

nmlkj more than 25 years

Section A: Design Work

A3. The complexity of a project is sometimes measured by its budget. Please check all boxes which reflect your involvement in projects of differing budgets: Participant Lead Role Small budget (less than $20,000) gfedc gfedc

Small to medium budget (from $20,001 to $75,000) gfedc gfedc

Medium budget (from $75,001 to $200,000) gfedc gfedc

Medium to large budget (from $200,001 to $500,000) gfedc gfedc

Large budget (over $500,000) gfedc gfedc

Section A: Design Work

A4. Please describe the web interface design work that you currently do. 5

6

Section B: Web Design Guidelines

In this section, I ask for your views about the role that web design guidelines might play in the design process.

B1. Have you used web design guidelines in your work?

nmlkj Yes

nmlkj No

Section B: Web Design Guidelines B2. Please list web design guideline sets you use, or have used, in your work. 5

6

Section B: Web Design Guidelines

B3. How did you learn about the web design guideline sets you use? Please select all that apply.

gfedc Blog gfedc Listserv or message board

gfedc Book gfedc Newspaper

gfedc Client gfedc Journal (online)

gfedc Consumer Magazine gfedc Journal (print)

gfedc Educational Institution gfedc Other Designer

gfedc Employer gfedc Podcast

gfedc e­newsletter gfedc Search engine

Other (please specify) 5

6

Section B: Web Design Guidelines

B4. Are there web design guidelines that you are aware of, but do not use?

nmlkj Yes

nmlkj No

If YES, please identify the sets and describe why you do not use them. 5

6

Section B: Web Design Guidelines B5. How frequently have web design guidelines helped you address design problems?

nmlkj Often

nmlkj Sometimes

nmlkj Rarely

nmlkj Never

nmlkj Don’t know

Section B: Web Design Guidelines

B5.1 Please describe HOW the web design guidelines helped. 5

6

Section B: Web Design Guidelines

B5.2 You answered never or rarely in the previous question, please describe why you think web design guidelines DID NOT help. 5

6

Section B: Web Design Guidelines

B6. A designer once said, “most web design guidelines are lacking key information needed to be effective.” Please share your opinions regarding the effectiveness of guidelines in solving web interface design problems. 5

6

Section B: Web Design Guidelines B7. Please indicate the extent to which you agree or disagree with the following statements: Strongly Strongly Agree Neutral Disagree Agree Disagree Guidelines are useful in helping to solve design problems. nmlkj nmlkj nmlkj nmlkj nmlkj

Guidelines are good at communicating solutions to design problems. nmlkj nmlkj nmlkj nmlkj nmlkj

Designers should consult guidelines to solve design problems. nmlkj nmlkj nmlkj nmlkj nmlkj

Designers must always follow guidelines. nmlkj nmlkj nmlkj nmlkj nmlkj

Section B: Web Design Guidelines

B8. Awareness of specific guideline sets varies greatly among designers. How familiar are you with any of the following guideline sets? I have applied I am not aware of I am aware of this I have read this guidelines from this set set set this set ISO 9241: Ergonomic requirements for office work with visual display nmlkj nmlkj nmlkj nmlkj terminals (VDTs)

Research‐based Web Design and Usability Guidelines (U.S. Dept. of nmlkj nmlkj nmlkj nmlkj Health and Human Services)

Research‐based Web Design and Usability Guidelines (National Cancer nmlkj nmlkj nmlkj nmlkj Institute)

Web Content Accessibility Guidelines, 1.0 (W3W) nmlkj nmlkj nmlkj nmlkj

Web Content Accessibility Guidelines, 2.0 (W3W) nmlkj nmlkj nmlkj nmlkj

Section C: Web Design Trade­offs

It has been suggested that interface design involves a series of design trade­offs. Trade­offs represent choices a designer must make between different design solutions. In this section, I ask for your views on these possible trade­offs.

C1. How often have you experienced web interface design trade­offs in your work?

nmlkj Often

nmlkj Sometimes

nmlkj Rarely

nmlkj Never

nmlkj Don’t know

Section C: Web Design Trade­offs C2. Please describe experiences you have had with web interface design trade­offs in your own work. 5

6

Section C: Web Design Trade­offs

C3. Have you experienced a trade­off between ENSURING USER ACCESSIBILITY and THE USE OF IMAGES AND MULTIMEDIA?

nmlkj Yes

nmlkj No

nmlkj Uncertain

If YES, please describe 5

6

ENSURING USER ACCESSIBILITY guidelines are intended to ensure that all users can access a website, potentially using assistive technologies. USE OF IMAGES AND MULTIMEDIA guidelines generally focus on the use of graphics and interactive media in the design of a website.

Section C: Web Design Trade­offs

C4. Have you experienced a trade­off between ENSURING USER ACCESSIBILITY and EMPLOYING SCREEN BASED CONTROLS / WIDGETS?

nmlkj Yes

nmlkj No

nmlkj Uncertain

If YES, please describe 5

6

ENSURING USER ACCESSIBILITY guidelines are intended to ensure that all users can access a website, potentially using assistive technologies. EMPLOYING SCREEN BASED CONTROLS/WIDGETS refers to the use of radio buttons, check boxes, drop­down lists and auto­tabbing functionality.

Section C: Web Design Trade­offs C5. Have you experienced a trade­off between OPTIMIZING USER EXPERIENCE and DISPLAYING IMAGE AND TEXT RICH CONTENT?

nmlkj Yes

nmlkj No

nmlkj Uncertain

If YES, please describe 5

6

OPTIMIZING USER EXPERIENCE guidelines encourage the efficient and effective interaction between the user and the web site. DISPLAYING IMAGE AND TEXT RICH CONTENT guidelines refer to choices about using scrolling or multiple pages to display content.

Section C: Web Design Trade­offs

C6. Have you experienced a trade­off between ENSURING A WEB PAGE IS LAID OUT CLEARLY and DISPLAYING DENSE or COMPLEX CONTENT?

nmlkj Yes

nmlkj No

nmlkj Uncertain

If YES, please describe 5

6

ENSURING A WEB PAGE IS LAID OUT CLEARLY is intended to ease comprehension. Guidelines refer to the placement of content in a page and the use of space and contrast in the page design. DISPLAYING DENSE or COMPLEX CONTENT guidelines refer to the organization of content (grouping, acknowledging multiple audiences, enabling quick understanding of content).

Section D: Tell me about yourself

In this section, I ask for information about yourself to help me interpret the findings. D1. How many personnel does your company employ?

nmlkj I am self employed

nmlkj 2 to 10

nmlkj 11 to 50

nmlkj 51 to 100

nmlkj 101 to 500

nmlkj 501 to 1000

nmlkj over 1000

Section D: Tell me about yourself

D2. Please indicate the extent to which you agree or disagree with the following statements: Strongly Strongly Agree Neutral Disagree Agree Disagree My organization encourages the use of web design guidelines. nmlkj nmlkj nmlkj nmlkj nmlkj

My organization ensures that I have access to web design guidelines. nmlkj nmlkj nmlkj nmlkj nmlkj

My organization requires the use of web design guidelines. nmlkj nmlkj nmlkj nmlkj nmlkj

Section D: Tell me about yourself

D3. How many other web interface designers, if any, work at your company?

nmlkj None

nmlkj 1

nmlkj 2­4

nmlkj 5­9

nmlkj 10­14

nmlkj 15 or more

Section D: Tell me about yourself D4. How long have you worked in your current position?

nmlkj Less than 1 year

nmlkj 1­3 years

nmlkj 4­6 years

nmlkj 7­10 years

nmlkj 11­15 years

nmlkj 16­20 years

nmlkj over 20 years

Section D: Tell me about yourself

D5. Please list your certifications, diplomas, or degrees, if any, and indicate the areas of specialization: 5

6

Section D: Tell me about yourself

D6. What is your country of residence? 6

Section D: Tell me about yourself

D7. Please select your age range.

nmlkj Under 25

nmlkj 25­34

nmlkj 35­44

nmlkj 45­55

nmlkj Over 55

nmlkj I prefer not to answer

Section D: Tell me about yourself D8. Please select your gender.

nmlkj Female

nmlkj Male

nmlkj I prefer not to answer

Section D: Tell me about yourself

D9. I welcome any additional comments regarding web design guidelines. 5

6

Section D: Tell me about yourself

Along with this questionnaire, I will be interviewing web interface designers. If you are willing to lend your time for a 20­30 minute phone interview, please indicate below.

I am willing to participate in a 20­30 minute phone interview

nmlkj Yes

nmlkj No, thank you.

If you agree to participate in an interview (and provide contact information), I will review your answers to this questionnaire prior to the interview.

If you would rather not participate in an interview, your answers will remain anonymous.

Contact Information

Contact Information (for the 20­30 minute phone interview)

Name:

Email Address:

Phone Number:

Thank you

Please reconfirm your consent for your responses to be included in the study.

nmlkj YES, include my responses in the study

nmlkj NO, please remove my responses Thank you for taking the time to participate in this study.

If you have any questions, feel free to contact me at [email protected].

Please click Submit Questionnaire to finish. You will then be taken to my web page, where the aggregate results of the survey will be posted in September, 2010. Appendix H: Feedback form for pretest of questionnaire (Study #3)

The next two pages (273-274) were created directly in Adobe Acrobat (PDF) and distributed as electronic documents.

Appendices 272

Appendix I: Recruitment Messages (Study #3)

Recruitment Message A

Recruitment Pool: ACM SIGCHI membership lists Delivery Method: Email via listserv

Initial Email

SUBJECT: Seeking participants: share your thoughts about web interface design

EMAIL BODY: We’re seeking your help in a research study which looks at the role of design guidelines and design tradeoffs in the process of web interface design. We would be grateful if you could share your opinions and experiences in a web based questionnaire. The questionnaire takes roughly 20 minutes to complete. All of your responses will remain confidential.

This study is part of doctoral research in the Faculty of Information (iSchool) at the University of Toronto. We are interested in the experiences you and other web interface designers have with aspects of the design process, such as the use of guidelines and your experiences (if any) with design tradeoffs.

In appreciation for your participation, aggregate results of the survey will be posted at http://www.szigeti.ca after August 1, 2010. If you would like more information about this survey, please don’t hesitate to contact the lead researcher ([email protected])

To begin the survey, please click here.

Thank you very much for your time and your help.

Steve Szigeti PhD Candidate University of Toronto

Appendices 275

First Reminder Email (sent one week after the first email)

Recruitment Pool: ACM SIGCHI membership lists Delivery Method: Email via listserv

SUBJECT: Seeking participants: share your thoughts about web interface design

EMAIL BODY: Thanks to everyone who has already completed the questionnaire.

If you intended to participate, but have not yet done so, please consider this reminder. Your opinions as a web interface designer are important and valued. Your participation is voluntary, but we would be most grateful for your assistance in completing an web based questionnaire. The questionnaire takes roughly 20 minutes to complete. All of your responses will remain confidential.

This survey is part of doctoral research in the Faculty of Information (iSchool) at the University of Toronto. We are interested in the experiences you and other web interface designers have with aspects of the design process, such as the use of guidelines and your experiences (if any) with design tradeoffs.

Please know that you can refuse to answer any question, or withdraw at anytime. If you do not wish to respond to particular questions, please skip over them.

In appreciation for your participation, aggregate results of the survey will be posted at http://www.szigeti.ca after August 1, 2010. If you would like more information about this survey, please don’t hesitate to contact the lead researcher ([email protected])

To begin the survey, please click here.

Thank you very much for your time and your help.

Steve Szigeti PhD Candidate University of Toronto

Appendices 276

Second Reminder Email (sent the day before the survey closes)

Recruitment Pool: ACM SIGCHI membership lists Delivery Method: Email via listserv

SUBJECT: Follow up: Seeking participants: share your thoughts about web interface design

EMAIL BODY: Again, we would like to thank everyone who already completed the questionnaire and helped with this research project.

Since participation in the survey is confidential, we have no way of knowing if you have chosen to not participate. If this is the case, please forgive us this final email.

If you had intended to participate, but have not done so, please consider this reminder. Your opinions as a web interface designer are important and valued.

This survey is part of doctoral research in the Faculty of Information (iSchool) at the University of Toronto. We are interested in the experiences you and other web interface designers have with aspects of the design process, such as the use of guidelines and your experiences (if any) with design tradeoffs.

Please know that you can refuse to answer any question, or withdraw at anytime. If you do not wish to respond to particular questions, please skip over them.

In appreciation for your participation, aggregate results of the survey will be posted at http://www.szigeti.ca after August 1, 2010. If you would like more information about this survey, please don’t hesitate to contact the lead researcher ([email protected])

To begin the survey, please click here.

Thank you very much for your time and your help.

Steve Szigeti PhD Candidate University of Toronto

Appendices 277

Recruitment Letter (posting)

Recruitment Pool: IxDA (Interaction Design Association) discussion board Delivery Method: Discussion Board posting

Posting SUBJECT: Seeking participants: share your thoughts about the web interface design

EMAIL BODY: I’m posting to ask for your help in a study regarding web interface design process.

This survey study is part of doctoral research in the Faculty of Information (iSchool) at the University of Toronto. I am interested in experiences web interface designers have with aspects of the design process, particularly the use of web design guidelines and your experiences (if any) with design tradeoffs.

The study involves the completion of an online questionnaire, which takes roughly 20 minutes to complete. Your participation is voluntary, but I would be grateful for your assistance. All of your responses will remain confidential.

In appreciation for your participation, I will be sharing the aggregate results of the survey with participants. They will be posted at http://www.szigeti.ca in August, 2010.

If you would like more information about this survey, please don’t hesitate to leave a comment on this post or contact me directly at [email protected]

To begin the survey, please click here.

Thank you very much for your time and your help.

Steve Szigeti PhD Candidate University of Toronto

Appendices 278

Appendix J: Interview Script (Study #4)

Phone Interview Session Protocol (guide).

Pre Session Script

Thank you for volunteering to take part in a phone interview. You received a description of my study and you returned the consent form via email. Before we get started, do you have any further questions?

Respond to questions, if any.

Thank you for participating. I am interested in your opinions and thoughts about the process of web interface design, specifically design guidelines and design tradeoffs. Your responses will help us understand the awareness and use of guidelines, as well as the design tradeoffs that come into play during the design process.

I will be guiding the discussion by asking general questions.

I’d like to reiterate that all of your responses will be kept confidential. The audio recording and any notes taken by the interviewer will be destroyed five years after the dissertation defense. The data collected will not be used for any purpose other than those related to this study.

You may decline to answer any question, and you may withdraw at any time.

The session should from 20 to 30 minutes.

Before we begin, I would like to remind you to speak loudly and clearly as I will be taping our conversation. Thank you. We’ll start now.

Start recording

Appendices 279

Session Script

1. My first question is a general one: can you describe your role in the most recent project you completed?

A. Design Guidelines

[If they did not check NO to the use of design guidelines in their work]

2. Did you use any web design guidelines in your most recent project? [If they did use one, go to 2.A..1. If they did not, go to 2.B.1.) 2.A.1 If so, which guideline set did you use? 2.A.2 Do you recall if the guideline set was helpful? Tell me about that.

2.B.1 If not, why did you not use a guideline set?

3. In the questionnaire were a number of statements, and you were asked to indicate your level of agreement or disagreement. In response to the statement, “My organization encourages the use of web design guidelines” you indicated that you agreed/disagreed. Why do you think they would/would not encourage guideline usage?

4. Would you say that there is a positive or negative view towards the use of web design guidelines in your organization? 4.A Why do you think that view exists?

5. I noticed that you described how you’ve used guidelines in your work. Could you elaborate on where in the design process you used guidelines (was it in the initial stages or latter stages or at point in between)?

6. Describe how you first use a guideline set? I’m thinking here of the process of first considering a set of guidelines.

Appendices 280

7. Please describe the format of the guideline sets you’ve recently used. 7.A. How are they presented? 7.B. Do you prefer that format over other formats? Why or why not?

8. It has been said that interface design is art and should not involve the application of principles. Do you agree with that statement? 8. A. Please elaborate and share your opinion regarding the role of guidelines in the design process.

B. Design Tradeoffs I now want to ask about design problems and tradeoffs during the design process. It has been suggested that interface design involves a series of design tradeoffs. Tradeoffs represent choices a designer must make between different design solutions.

9. In the questionnaire you indicated that you often/sometimes/rarely/never experience any design tradeoffs in your work. [ If often or sometimes, go to 9.A If rarely or never, go to 9.B]

9.A Are the design tradeoffs you face unique to each project, or do you find similar tradeoff’s occur in various projects? Please elaborate.

9.A.2 Do you address design tradeoffs in the same manner for each project, or does the design decision you make change from project to project? Why? 9.A.3 In your experience as a designer, do you believe guidelines would help you find solutions to design tradeoffs?

9.B. Why do you think you’ve rarely/never experienced design tradeoffs in your work?

10. I’ve asked a few questions about design problems. Solving design problems is a potential and necessary part of the design process. How many hours per day do you estimate are devoted to solving a design problem?

Appendices 281

11. Do you believe that using guideline sets might reduce the time you spend solving such problems?

C. Knowledge If seems that guidelines and advice on design is changing all the time and I know that it’s difficult to keep up to date. The next few questions ask about your awareness of resources to address design problems.

12. Who or what do you consult if you have a concern about a web interface design?

13. Please describe how you share web design knowledge with other designers.

14. Would you find access to more web design guideline sets helpful? Why or why not?

15. Some web design guidelines are drawn from expert advice – the experience of individuals involved in design. Other guidelines are drawn from published research. Do you believe there is a difference in the validity of either type (expert advice or published research)? Why or why not?

D. Wrap-up

16. Do you have any further thoughts or ideas on the use of guidelines in web interface design?

17. Is there anything you would like to add, or comment on, that has not been covered?

Post Session Script

That is the end of the session. Thank you very much for participating in this study. Stop Recording. Thank you for your time. You have my email address should you need to contact me for any reason.

Appendices 282

Appendix K: Interviewees’ Information and Consent Form Email (Study #4)

Email Subject Line: Informed Consent for Study Participation: Perceptions and use of web design guidelines

Email Body Copy: Thank you for agreeing to lend 20 to 30 minutes of your time for a phone interview. Your help and participation is appreciated. During the phone interview, I will ask you more questions about your opinions regarding guidelines, design tradeoffs and how you share web interface design knowledge.

Please reply to this email,

1. With dates and times when you would be available for a 20-30 minute phone interview, and, 2. By including “I agree” in the body copy if you agree to participate in this study and agree with the following statements:

 I have freely volunteered to participate in this study.  I agree to having this session audio recorded.  I have been informed in advance about the nature of the study and what procedures will be followed.  I have been given the opportunity to ask questions and have had my questions answered to my satisfaction.  I understand that the information I provide will be treated confidentially.  I understand that the information I provide will not be used for any purposes other than those related to this study.  I am aware that I have the right to decline to answer any questions, and may withdraw consent and discontinue participation at any time. If I withdraw, my responses will be erased and removed from the study.

A summary of the research results will be available at http://www.szigeti.ca in August, 2010.

There are no known or anticipated risks to participants.

If you have any questions or concerns either now or during the course of the study, please address them to the researcher, Steve Szigeti (416-XXX-XXX) or his supervisor, Prof. Joan Cherry (416- XXX-XXX). If you have any questions regarding your rights as a participant, you may contact the Ethics Review Office at [email protected] or 416-946-3273.

Steve Szigeti PhD Candidate, University of Toronto URL: http://www.szigeti.ca email: [email protected] mobile: 416-XXX-XXXX

Appendices 283