Dimensions and Elements of People's Mental Models of an Informationrich

Dimensions and Elements of People’s Mental Models of an Information-Rich Web Space

Yan Zhang School of Information, University of Texas at Austin, 1616 Guadalupe, Austin, TX 78701. E-mail: [email protected]

Although considered proxies for people to interact with Zheng, & Gonzalez, 2006; Kudikyala &Vaughn,2005; Rode, a system, mental models have produced limited practi- Rosson, & Perez-Quiflones, 2004; Young, 2008). cal implications for system design. This might be due to After two decades of research, however, the impact of men- the lack of exploration of the elements of mental models resulting from the methodological challenge of measur- tal models research on system design and user training is still ing mental models. This study employed a new method, limited (Roger, 1992). One possible reason is the lack of concept listing, to elicit people’s mental models of an empirical research on the elements, components, or dimen- information-rich space, MedlinePlus, after they interacted sions of people’s mental models of systems. Nevertheless, with the system for 5 minutes. Thirty-eight undergradu- due to their inherent abstractness, eliciting and represent- ate students participated in the study.The results showed that, in this short period of time, participants perceived ing mental models has been a challenge (Borgman, 1986). MedlinePlus from many different aspects in relation to Popular methods such as semistructured interviews, concept four components: the system as a whole, its content, mapping, repertory grid technique, and pairwise rating tended information organization, and interface. Meanwhile, par- to limit the emergence of new dimensions by imposing spe- ticipants expressed evaluations of or emotions about cific questions or concepts on users. Drawing, another widely the four components. In terms of the procedural knowledge, an integral part of people’s mental models, only used method, is limited in revealing people’s understand- one participant identified a strategy more aligned to the ing of implicit system processes, such as search mechanisms capabilities of MedlinePlus to solve a hypothetical task; (Zhang, 2008a). the rest planned to use general search and browse strate- This study presents an attempt to elicit elements and gies. The composition of participants’ mental models of dimensions of mental models using a bottom-up approach. MedlinePlus was consistent with that of their models of information-rich Web spaces in general. Specifically, we explored people’s mental models of an information-rich Web space, MedlinePlus, after they interacted with the system for 5 minutes. Information-rich Web spaces are hyperlink-based information systems that contain Introduction a large amount of information, often thousands or millions of Mental models are knowledge structures that people Web pages, in a particular domain or across domains, such employ to represent, make sense of, and interact with the as Wikipedia and aclu.org. Information-rich spaces serve a external world (Gentner & Stevens, 1983; Johnson-Laird, wide range of audiences and the information could be in var- 1983). Researchers in human computer interaction (HCI) and ious forms, such as text, image, and videos (Shneiderman, information retrieval (IR) postulate that people form men- 1997). As the Web keeps expanding, more and more gov- tal models of a computer system that they interact with to ernment agencies, military organizations, and companies are understand “what the system contains, how it works, and looking to such hypermedia-based solutions for organizing why it works” (Carroll & Olson, 1988, p. 51; Sasse, 1991). and disseminating information (Farris, Jones, & Elgin, 2002). Therefore, knowledge about mental models is expected Studies suggested that people often quickly made deci- to provide insights into the design of computer systems sions whether they were going to stay on a Website (Sillence, that are easy to use, intelligent in supporting users, and Briggs, Harris, & Fishwick, 2007) and they tended to spend effective in reducing unnecessary human errors (Graham, less than 5 minutes on a particular Website (Burk, Martin, Reilly, & Kuperman, 2003; Spink & Cole, 2006). Some software developers contended that if common-use software or Received April 7, 2010; revised June 13, 2010; accepted June 14, 2010 Websites cannot make sense to the user in the first 5 min- © 2010 ASIS&T • Published online 2 August 2010 in Wiley Online Library utes, they are losing customers. Thus, it is important to learn (wileyonlinelibrary.com). DOI: 10.1002/asi.21406 what kind of mental models people develop within such a

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY, 61(11):2206–2218, 2010 short period of time. So in this study we limited participants’ such as data content, interactive nature of the system, exis- interaction with MedlinePlus to 5 minutes. MedlinePlus was tence of multiple files, and use of controlled vocabulary. It is selected as the platform for the study due to the fact that apparent that both approaches could limit the emergence of consumer health information-seeking has become one of the new dimensions and elements (Wilson & Rutherford, 1989). most popular activities online (Fox, 2005). Web-based con- The more structured interviews tend to scaffold subjects’ sumer health information systems, like MedlinePlus, deserve mental models by limiting the aspects that they could talk more research. about. The less structured interviews were less constraining; however, mental models were often reported in a predefined structure, rather than being allowed to emerge from the data Related Literature naturally. Researchers in Information Science (IS) have examined A second widely used method to elicit mental models was people’s mental models of various IR systems, includ- asking users to identify the type and/or strength of relation- ing online library catalogs (Borgman, 1986; Dimitroff, ships between a set of concepts (e.g., Hanisch, Kramer, & 1992; Kerr, 1990; Slone, 2002), news databases (Katzeff, Hulin, 1991), or draw a concept map of a set of concepts 1990), Web search engines (Efthimiadis & Hendry, 2005; (Chang, 2007). For example, Langan-Fox, Platania-Phung, Muramatsu & Pratt, 2001), digital libraries (Makri et al., and Waycott (2006) used the pairwise ratings method to elicit 2007), experimental IR systems (Cool, Park, Belkin, subjects’ mental models of a mobile phone network. The Koenemann, & Ng, 1996; Savage-Knepshield, 2001), visual- authors created a set of 15 concepts from the manual and ization interface (Chen & Czerwinski, 1997), 3D virtual envi- asked participants to rate the relatedness of pairs of concepts ronments (Billinghurst & Weghorst, 1995), the Internet or the concerning network services on a 7-point scale (1 = not at all Web (Bruce, 1999; Thatcher & Greyling, 1998), and college related, 7 = highly related). The matrices generated by the Websites (Otter & Johnson, 2000). Although information- ratings were then used to produce an associative network of rich Web spaces are an important type of Web-based IR the concepts to represent participants’ mental models of the system and have been gaining popularity in recent years phone network. In such studies a limited number of concepts (Shneiderman, 1997), few studies have examined people’s can be presented to users. Thus, these methods also tend to mental models of such systems. limit the emergence of new dimensions or elements of mental Due to the abstractness of mental models, early research models (Royer, Cisero, & Carlo, 1993). mainly focused on identifying characteristics of mental mod- The other widely used method to represent mental models els, such as simple, unscientific, and parsimonious (e.g., is drawing. In the drawings, participants were often asked to Norman, 1983) or examining the impact of mental mod- draw their perceptions about a system (e.g., Efthimiadis & els on people’s system use behavior and performance (e.g., Hendry, 2005; Zhang, 2008b). Drawings are illustrative in Borgman, 1986; Halasz & Moran, 1983; Kieras & Bovair, demonstrating the elements included in mental models. For 1984). Little effort was made to explore the elements, dimen- example, based on the elements presented in drawings, sions, or composition of mental models. As researchers in Thatcher and Greyling (1998) identified people’s mental human factors, HCI, and IR seek means to apply mental mod- models of the Internet into six categories: (1) interface and els to the design of system and instructional materials, the utilitarian functionality, (2) central database, (3) user to the importance of understanding the content of mental models is world, (4) simple connectivity, (5) simple modularity, and becoming paramount (Young, 2008). (6) modularity and networking. Papastergiou (2005) identi- In the existing literature, one of the main methods used to fied eight categories of high school students’ mental models elicit people’s mental models was interviews (Rutherford & of the Internet: (1) nondigital entity, such as information and Wilson, 1992; Sasse, 1991; Staggers & Norcio, 1993). In users without computers involved, (2) services and content, some studies, participants were asked to answer a set of (3) users’ computers, (4) servers, (5) connection between questions concerning a system. For example, Zhang (2008a) two computers, (6) few computers linked through a con- investigated undergraduate students’ mental models of the nection point, (7) computer network, and (8) network of Web as an IR system by asking their perceptions of four main computer networks. However, drawings fall short in repre- components of general IR systems: the information space, the senting invisible or implicit system processes, such as query information organization, search mechanism, and interface. processing and relevance ranking mechanisms (Papastergiou, Papastergiou (2005) examined subjects’mental models of the 2005; Thatcher & Greyling, 1998; Zhang, 2008a,b). Internet by asking them to describe their perceptions regard- To better inform system design and user instruction, it is ing the Internet as an entity, a centralized or a distributed necessary to understand mental models from the end user’s system, and the structure of the Internet. perspective. Given the limitations of the current methods, new Some interviews were less structured. For example, methods are needed to allow the elements and dimension Dimitroff (1992) asked subjects to describe similarities and of people’s mental models to emerge without any preim- differences between card catalog and an OPAC system posed structure. It is also necessary to be able to elicit and describe how the OPAC system worked. Subjects’ men- and represent people’s procedural knowledge of using the tal models were then evaluated against a predefined system system, because such knowledge is an integral part of men- conceptual model that included eight discrete components, tal models (Brandt & Uden, 2003; Carroll & Olson, 1988;

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—November 2010 2207 DOI: 10.1002/asi deKleer & Brown, 1988; diSessa, 1986; Gott, Benett, & directories of health professionals and hospitals, news feeds, Gillet, 1988; Nielsen, 1990). and tutorials. In MedlinePlus, users can access information by browsing subject hierarchies or alphabetical lists. Search was supported, but the search function was rather simple. Research Questions There was no advanced search option and users cannot limit the search to certain fields. Mental models do not have firm boundaries (Norman, 1983). Users often come to an IR system with a set of knowledge or assumptions (Bruce, 1999). Thus, we are also Measuring Mental Models interested in participants’ initial model of information-rich Students’ initial model of information-rich Web spaces Web spaces in general. In the study, two research questions was elicited in the demographic questionnaire by two are explored: requests: (1) “Please describe characteristics of information- rich Web spaces based on their experience and understand- 1. What is participants’ mental model of information-rich ing,” and (2) “Please describe your general approaches to use Web spaces before they interact with MedlinePlus? 2. What is their mental model of MedlinePlus, a typical this type of Web spaces.” information-rich Web space, after they interact with the Students’ mental models of MedlinePlus were measured system for 5 minutes? by three methods, a concept listing protocol, a semistructured interview, and a drawing task. In the concept listing protocol, participants were asked to list concepts related to Methods MedlinePlus in the order that the concepts appeared in their mind. The concept listing lasted for 5 minutes. Every list of Participants concepts was viewed as the result of a participant’s cogni- A total of 39 (19 males, 20 females) undergraduate stu- tive process of making sense of the system. By this means, dents at the University of North Carolina (UNC) participated no predefined structure was imposed and participants’ men- in the study. The participants had never used Medline- tal models of MedlinePlus were allowed to emerge from the Plus before the study. Because spatial ability was related data. A Web-based graphic interface was designed to assist to people’s navigation in hyperlink-based systems (e.g., participants in inputting concepts. The submitted concepts Campagnoni & Ehrlich, 1989), one participant was excluded were stored in a MySQL database. from the data analysis due to his extremely low spatial abil- Mental model theory suggests that people’s mental mod- ity. The ages of the subsequent 38 participants (18 males, els of a system also encompass procedural knowledge of 20 females) ranged from 18 to 22, with an average age of using the system (e.g., Fein, Olson, & Olson, 1993; Mayer, 21 years old (SD = 1.07). 2002). It was foreseen that the concept listing protocol might To control the impact of domain knowledge on mental have a limited capacity to elicit the procedure knowledge. model construction, students who majored in medical-related Therefore, semistructured interviews were performed. In the fields, such as nursing, medicine, pharmacy, public health, interviews, participants were presented with a hypothetical and biology, were excluded from the recruitment. As a result, task, as shown below, and asked to describe steps that they participants in the study were from 22 different nonmedical- would follow to solve the task. related majors, such as art history, business, English, Spanish, communication studies, political science, physics, geogra- Imagine that you are required to write a paper about hepatitis phy, and math. One subject had not decided on a major area to arouse public attention to the seriousness of this disease. In the paper, you want to include, but are not limited to, aspects of study. such as what is the difference between chronic and nonchronic hepatitis? What determines this? What are the differences The Platform: MedlinePlus between hepatitis A, B, and C, and what are the treatments for each of them, and can we prevent the development MedlinePlus (www.medlineplus.gov) provides authorita- of liver cirrhosis among patients with chronic hepatitis? tive medical and health information for the general public. You decide to use MedlinePlus to collect information for It was launched in 1998 by the National Library of Medicine your paper. (NLM) in response to the intensive use of the MEDLINE database by general consumers via theWeb. Two sources con- They were also asked to describe their impressions of tribute to the collection of MedlinePlus: National Institutes of MedlinePlus concerning its collection, search mechanisms, Health’s (NIH) publications of consumer health information information organization, and the interface. Two exam- based on the medical research that NIH sponsors, and publi- ples of the interview questions are: “What information is cations of professional medical societies and voluntary health provided by MedlinePlus,” and “How do you think infor- agencies without commercial or business motives (Lindberg, mation in MedlinePlus is organized?” The interviews were 2000; Marill, Miller, & Kitendaugh, 2006). MedlinePlus audiorecorded. After the interviews, subjects were asked to also licenses information sources, such as medical dictio- draw their perceptions of MedlinePlus. The drawings will not naries, encyclopedia, information about prescription drugs, be analyzed and reported here.

2208 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—November 2010 DOI: 10.1002/asi Procedure a particular aspect of a component. For example, the code for “well organized” indicates that it is an evaluation of the Participants were scheduled to take part in the study at information organization in MedlinePlus. The author coded their convenience. Upon arrival, they were welcomed and all the concepts and a second coder coded 10% of the data. were provided a brief introduction to the study. After signing The intercoder agreement (Krippendorff’s α) was 78.4% in a consent form, participants were asked to finish a 2-page the first round of coding (Krippendorff, 2004). A review ses- 20-item ETS VZ-2 test (Eckstrom, French, Harman, & sion between the two coders revealed that the majority of the Derman, 1976); each page was limited to 3 minutes. Partic- disagreements appeared at the second level of the code. For ipants then finished a demographic questionnaire, reporting example, the second coder initially coded “NIH” as “system: their age, major field of study, computer experience, experi- similar sites.”After the discussion, she agreed with the author ence with health and medical information searching online, and coded it as “system: agencies involved.”After the review, and their perceptions of information-rich Web spaces. Upon the intercoder reliability reached 95.7%. the completion of the questionnaire, instead of watching a The audiotaped semistructured interviews were tran- prerecorded video introduction to the system, the participants scribed. The transcripts were imported into QSR N6 software. were instructed to spend 5 minutes to explore MedlinePlus Distinct strategies of solving problems using MedlinePlus in the way that they normally do when freely exploring a were identified by reading participants’ descriptions of the new information system. After the exploration, participants steps that they planned to take to solve the hypothetic task. finished the set of protocols measuring their mental models The rest of the transcripts were mainly used as a source to of the system, including a concept listing, a semistructured assist the interpretation and verification of certain concepts interview, and a drawing task. in the concept listing protocol. Data Analysis Results The demographic data were analyzed using descriptive statistics. Descriptions that participants provided in the demo- Experience With Health Information Searching graphic questionnaire about their perceptions of information- The 38 participants had used the Web for 6 to 13 years rich Web spaces were imported into QSR N6 software. Open (mean = 9.9; SD = 1.83). Among them, 35 (92.1%) had coding was employed to code the data. Whenever an ele- used the Web for health information. Table 2 lists the ment, such as videos, Q&A, and database or a strategy of resources that participants used. As shown in the table, there using the spaces, such as search, browse, and Google to were three main resources for health information: people, research the site appeared, it was coded into a category. The Websites, and books. About 79% of the participants reported subsequent categories were organized from bottom-up based that they sought medical information from family, friends, on what component of the systems the categories referred and doctors. One mentioned that she asked for medical advice to. A second coder coded the descriptions and the intercoder from her athletic trainer. The most popular online sources for agreement was 91.6%. health information were general search engines,WebMD,and The concepts that participants contributed in the concept Wikipedia. About 29% of the participants had also used the listing protocol was exported to an Excel sheet. Open coding online health sources provided by UNC (i.e., health infor- was employed to code the data.The coding unit was a concept. mation under the unc.edu domain). Only 13.2% used books When multiple concepts were involved in an entry, multi- for health information. Other resources used by only a couple codes were assigned to the entry. A coding schema was ple of participants included Yahoo Health, PubMed, and the developed by coding a subset of the concepts. The schema People’s Pharmacy. contained the names of categories emerging from the coding, definitions of the categories, and examples for each cate- Participants’Initial Model of Information-Rich gory. The scheme was then applied to code the rest of the Web Spaces concepts. Several coding examples are provided in Table 1. The first level of the code refers to major components of the Participants’initial model of information-rich Web spaces MedlinePlus system and the second level of code refers to was induced from their descriptions of their general

TABLE 1. Coding examples. TABLE 2. Information sources for health and medical information.

Concepts contributed Information sources Total by subjects Assigned codes General search engines 32 (84.2%) Lung cancer Content: speciﬁc Family & friends 30 (78.9%) Alphabetical Information organization: schema Doctors 30 (78.9%) Well organized Information organization: evaluation WebMD 24 (63.2%) Tabs Interface: element Wikipedia 18 (47.4%) Search Interface: function UNC online health sources 11 (28.9%) CDC System: agencies involved Books 5 (13.2%)

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—November 2010 2209 DOI: 10.1002/asi perceptions about and strategies of using these spaces. The expressed their evaluations of or emotions about each com- analysis showed that participants represented information- ponent. For example: rich Web spaces in relation to four system components: • system, content, information organization, and interface. System: convenient, accessible • Content: a lot of info, useful, helpful, reliable, readable, The system component was participants’ representation of updated frequently, clear, informative, unreliable, over- the overall systems, rather than any individual part of them. whelming This component was represented in relation to two aspects: • Information organization: rigid organization the structure of and the audience for the systems. For exam- • Interface: easy to navigate, cluttered, distracting ple, in talking about the structure, one subject suggested that: In describing their approaches of using information-rich Web spaces, the participants listed the following strategies:

Based on my own experience and understanding, information- • Google to reach the site rich Websites are defined as those that compile any amount • Search of databases into what could be considered a data warehouse. • Browse Utilizing the warehouse search capabilities, users are directed • Read citation pages to the individual databases, which are hyperlinked onto • Click random links the site. • Avoid watching videos • Glance for graphs, pictures, and headings In talking about the audience, participants thought that some information-rich Web spaces served novices in the field In summary, participants had an initial model of of their interest and some were directed more toward experts. information-richWeb spaces. In the initial model, two distinct A couple of participants suggested that such spaces were but closely related dimensions emerged: (1) representa- useful for a wide variety of people. tions: participants represented information-rich Web spaces The content component was participants’ representation in relation to four components: system, content, information of the information contained in information-rich Web spaces. organization, and interface; and (2) evaluations: participants This component was perceived in relation to three aspects: simultaneously expressed evaluations of or emotions about subject, type, and format. For the subject, some partici- the four components. They also had a set of general strategies pants described information-rich Web spaces as covering one or heuristics of using such spaces. particular topic, while some said that they covered various topics. For the type of information, participants mentioned An Overview of Participants’Mental Model Q&A, help, and advertisements. For the format, participants of MedlinePlus expected to see images, graphics, text, videos, multimedia, Participants’ mental model of MedlinePlus was derived and audios. from the analysis of the concepts that they contributed in The information organization component was participants’ the concept listing protocol. The result suggests that their understanding of the ways in which information is orga- mental model of MedlinePlus shared a similar composition nized. Generally, they recognized that, in information-rich to that of their initial model of information-rich Web spaces. Web spaces, information was interlinked; some links led They represented and evaluated MedlinePlus in terms of four to outside sources. Instead of having a specific informa- components: tion organization schema, such as alphabetical or hierarchical organization, participants tended to have only a very • System: the overall MedlinePlus rather than any part of it general idea that information was categorized in some way. • Content: the content included in MedlinePlus • For example, one participant commented that: Information organization: how information is organized in MedlinePlus • Interface: the interface of MedlinePlus Information is usually organized into categories of some kind on these Websites, to make it more accessible for users .... Table 3a,b, respectively, summarizes the aspects from The categories can be manually looked through usually. which each component was represented or evaluated in the two dimensions. The interface component was participants’ representation Informed by the mental model theory, participants’mental of the interface of information-rich Web spaces. In represent- model of MedlinePlus also encompasses procedural knowl- ing this component, participants thought about navigational edge of solving problems using the system. The procedural elements, including menus, side bars, tabs, hyperlinks, sub- knowledge was elicited from their descriptions of steps that headings, and bold font. They also mentioned functionali- they would employ to solve the hypothetical task in the ties made available through the interface, including search, semistructured interview. Three distinct strategies were pro- advanced search, suggesting other pages of interest, and posed. StrategyA is a general searching strategy: participants frequently searched links and topics. submit a search query to the system and review the search When representing information-rich Web spaces in rela- results. Strategy C is a general browsing strategy: partici- tion to the four components, participants simultaneously pants follow links from the homepage to the page where the

2210 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—November 2010 DOI: 10.1002/asi answer is located. Strategy B is a combination of searching B is more closely aligned to the capabilities of MedlinePlus and browsing, where participants search a disease or condi- and requires fewer clicks to reach the desired information. tions, click on the first result to reach the health topic page pertaining to the disease or condition, and browse the links Participants’Mental Model of MedlinePlus on the health topic page to find relevant information. Strategy In this section, participants’ mental model of MedlinePlus after they used the system for 5 minutes is reported in detail. TABLE 3a. Representations dimension of participants’ mental models of MedlinePlus. Representations and evaluations of the system component. System Content Info. org. Interface After a 5-minute interaction, participants represented Med- (76) (413) (35) (102) linePlus as an integrated system from six aspects. Table 4a – Structure – Subject – Information – Elements lists the concepts and the number of concepts associated with – System behavior – Type organization – Functions each aspect. – Agencies involved – Presentation schemas – Results In representing the structure of MedlinePlus, participants – Audience – Format thought that MedlinePlus was a large database and storage – Usage of the system – Similar sites tank. It had its own information, but also linked to external resources. The system worked primarily based on links rather The number in parentheses is the number of concepts that the participants than search features. In representing the system’s behavior, dedicated to each component. participants speculated that MedlinePlus aggregated, pooled, and filtered information from other medical Websites and it was frequently updated. They noticed that MedlinePlus TABLE 3b. Evaluations/emotions dimension of participants’mental mod- was associated with government agencies and some medical els of MedlinePlus. associations. The audience for system could be both general System Content Info. org. Interface people (e.g., everyday users, lay people, novices, parents, (47) (89) (15) (52) adults, and juveniles) and medical professionals. – Usefulness – Quality – Information – Look & feel After the brief interaction, participants also started think- – Attributes – Utility organization – Search ing about the potential usage of the site. Some suggested that – Usability – Attributes – Navigation the system was a starting point for research and could be – Quantity – Results used to gather information about particular conditions. Some – Specific sections thought that the system was a good counseling tool for peo- The number in parentheses is the number of concepts that the participants ple to learn to be healthy and to self-diagnose conditions. dedicated to each component. Nevertheless, some participants pointed out that they would

TABLE 4a. Participants’ representations of MedlinePlus as an integrated system.

Aspects Concepts

System structure (22) – Database; Storage tank – The site seems to work primarily based on links – Links to outside info; links to other medical sites rather than search features. – Linked pages – Inside information System behavior (13) – Pop-up windows – Pooling information – Redirection – Filter – Updated daily – Aggregate Agencies involved (11) – General: government agencies, partners, – Speciﬁc: NIH – medical associations Audience (17) – All genders/ages – Professionals – Laypeople – Patients – Everyday users – Parents – Juvenile – Novices – Someone with an interest in a particular condition – Older adults – Adults Usage of the system (11) – Learning how to be healthy – Counseling, consult – Starting point – Self-diagnosing – Information gathering – Used to see if any major health issues – Explore – Would also seek info from doc – Useful to research minor twinges Similar systems (2) – WebMD – PubMed

When similar expressions were used to describe one concept, only one expression is listed in the table.

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—November 2010 2211 DOI: 10.1002/asi TABLE 4b. Participants’ evaluations of MedlinePlus as an integrated system.

Aspects Concepts

Attributes of the system (12) – Search oriented; searchable – Large database; quite large in scope – Similar to other information-rich sites – Data-rich, Resourceful – Free Usefulness (10) – Helpful – Good place to seek initial advice – Wide array of use – Good for avoiding doctor – Less need for doctors – Not self sufﬁcient Usability of the system (24) – Accessible – Fast paced – Quick access; fast; prompt – No glitches – Easy to use – Not reactive – User friendly – Not clear how they pick links Public awareness (1) – Not well known to the public

When similar expressions were used to describe one concept, only one expression is listed in the table. still need to consult and seek information from doctors. It is TABLE 5a. Participants’ representations of the type, format, and presen- worth noting that several participants thought of two systems tation of information in MedlinePlus. that they were more familiar with, WebMD and PubMed. Aspects Concepts The evaluations or emotions about the system component were in relation to four aspects: attributes of the system, use- Type (50) – Dictionary – References for medicine – Directories – Tutorials fulness, usability of the system, and public awareness, as – Encyclopedia – Presentations shown in Table 4b. – FAQs – Scholarly articles Attributes of the system refer to participants’ understand- – Glossary – Medical sites ing of the characteristics of MedlinePlus as an integrated – Journals – Magazine system. In the study, participants described MedlinePlus as a – News – Q&A large and resourceful search-oriented system. It was similar Format (24) – Text – Flash to other information-rich Web sites and it was free. Con- – Videos/Movies – Multimedia – Images/Photos/Pictures – PDF cerning the usefulness of the system, participants felt that Presentation (15) – Demonstrations – Short articles MedlinePlus was helpful. It was a good place for initial med- – Details – Several sections ical advice and could help people avoid seeing the doctor. – Overview – Summaries Only one participant pointed out that the system was not – Diagrams self-sufficient. As shown in the table, about half of the evaluations of When similar expressions were used to describe one concept, only one expression is listed in the table. the system component were about the usability of Med- linePlus. Participants generally agreed that the system was user-friendly, fast to access, easy to use, and without notice- treatments, supplements, or drugs and were more specific able glitches. Only a couple of them pointed out that it was than the ones at the topical level. Examples included dia- not reactive and it was not clear how the links were selected. betes, exercise routines, cardiology, endocrine system, ear One participant commented that MedlinePlus was not well infection, tuberculosis, x-rays, and insulin. known to the public. Concepts that participants contributed to represent the type, format, and presentation of information in MedlinePlus Representations and evaluations of the content component. are listed in Table 5a. As shown in the table, participants rec- Participants represented the content of MedlinePlus in rela- ognized various types of medical and health information in tion to four aspects: subject, type, format, and presentation MedlinePlus, including a dictionary, directories, an encyclo- of information. The subject of the information in Medline- pedia, FAQs, a glossary of diseases, journals, medical news, Plus was represented at three different levels: general, topical, Q&A, references for medicine, and tutorials. The information and specific. Concepts at the general level (55 concepts) was in different formats, such as text, image, video, flash, and do not reveal the subject matter of the site and do not dif- multimedia. At the same time, participants paid attention to ferentiate MedlinePlus from other sites. Examples included different ways in which information was presented in the information, data, advice, articles, literature, and sources. system. They found that some articles were divided into sev- Concepts at the topical level (146 concepts) described the top- eral sections; some information was presented as overviews, icality of the information in the system. Examples included summaries, diagrams, demonstrations, or short articles, and medical information, diseases, symptoms, treatments, health sometimes details were provided. information, diagnosis, supplements, and drugs. Concepts at When representing the content in MedlinePlus, partici- the specific level (123 concepts) described specific diseases, pants also expressed their evaluations of or emotions about

2212 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—November 2010 DOI: 10.1002/asi TABLE 5b. Participants’ evaluations and emotions concerning the content of MedlinePlus.

Aspects Concepts

Quantity of information (11) – Information on a large number of topics, – Not many articles including lesser-known conditions – Limited – Overwhelming amounts of information; – Additional content is needed Lots of information Quality of information (13) – Academic, Scientific; Well-researched – Reputable – Clear – Reliable, trustworthy – Consistent – Authoritative – Thorough – Question of credibility Utility of information (21) – Helpful – Understandable – Useful – No information on brown recluse bite – Interesting – Additional content other than medical data might – Relevant make it more interesting, such as an article – Informative about health insurance – Easy to read – Self-explanatory Attributes of information (42) Comprehensiveness Depth of information – Comprehensive – Basic; General – Diverse – Common – Not restrictive to one area of medicine – Quick information – National and local information – Broad; Breadth not depth Currency – Specific – Current; Contemporary Language – Recent; Up-to-date – Available in different languages Objectivity – Words were mostly monochromatic – Facts Others – Popular topics – Concise Evaluation of specific sections – The general information on bodily systems and – More description on black widow bite of content (2) functions supplements and provides a foundation for the information on diseases.

When similar expressions were used to describe one concept, only one expression is listed in the table.

the quantity, quality, utility, attributes, and speciﬁc sections form opinions about speciﬁc sections of content in the system. of the content, as shown in Table 5b. For example, one commented that: Concerning the quantity of information, some participants thought that MedlinePlus contained a good amount The general information on bodily systems and functions sup- of information, while others thought that there were not plements and provides a foundation for the information on many articles and additional content, such as information diseases. about health insurance and more descriptions on certain topics. Concerning the quality of information, participants had mostly positive feelings. They felt that the content of Representations and evaluations of the information organi- MedlinePlus was not only thorough and clear, but also zation component. The information organization compo- consistent, reputable, and reliable. Only one participant ques- nent was participants’ representations of how information tioned the credibility of the information. Concerning the was organized in MedlinePlus. Table 6a lists concepts that utility of information, participants felt that the information participants contributed to represent this component. in MedlinePlus was not only useful (e.g., relevant, useful, Despite the lack of emphasis (35 concepts; 5.6% of all the and informative), but also usable (e.g., self-explanatory and concepts in the representations dimension), participants were easy to read). able to recognize various information organization schemas At the same time, participants developed opinions con- in MedlinePlus, such as by subjects, alphabetical (array of let- cerning various attributes of the content, such as comprehen- ters, listing, encyclopedic), by body systems (anatomy, body siveness, currency, objectivity, and depth. They commented parts), by gender, age, and other demographic information, that information in MedlinePlus was common, basic, and fac- and by generic and brand names of drugs. One participant tual. It covered a broad spectrum of medical topics and was noted that multiple facets were used to organize the informa- updated regularly. Some information was even available in tion and several others noticed the cross listing of relevant multiple languages. A couple of participants also began to categories.

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—November 2010 2213 DOI: 10.1002/asi TABLE 6a. Participants’ representations of information organization in TABLE 7a. Participants’ representations of the interface of MedlinePlus. MedlinePlus. Interface – Color – About MedLine – Categories; Array of topics; Grouped; Organized by topic and concepts elements (61) – Links – About us – All body systems; anatomy; body parts; body systems – Lines on Webpage – Contact us; – Alphabetical; Array of letters; Encyclopedic; Listing – Hospital and doctor Contact information – Health related issues depending on gender/age/other demographics listings – Banner – Multi-faceted – Sections – Tabs – Hierarchical – Homepage – Purple – Cross-listing – Images – Ads – Drugs are listed under both generic and brand names – Search bar, Search box – Disclaimer – Subject headings – Logo When similar expressions were used to describe one concept, only one – Suggested articles – Menus expression is listed in the table. – Related illness – Questions and Answers – Links to related topics – Interface TABLE 6b. Participants’ evaluations of information organization in Med- – Title – Layout linePlus. Functions (39) – Search – Symptom finder – Local search – Pill identifier – Clearly listed – Browse – Good multimedia structure – Logical Results (2) – Search results yield – Suggested results – Simplified categories snippets – Well organized – Provides a comprehensive flow map to information When similar expressions were used to describe one concept, only one – Not enough subgroups expression is listed in the table.

When similar expressions were used to describe one concept, only one expression is listed in the table. When representing results, participants noticed the snippet that appears at the top of the result list and suggested results. Participants’evaluations or emotions concerning this com- Simultaneously, participants expressed their evaluations ponent were in relation to three aspects: look and feel, of information organization (15 concepts) in MedlinePlus, as navigation, and search, as shown in Table 7b. shown in Table 6b. Look and feel refers to participants’ evaluations of the Participants formed largely positive feelings about the visual appearance of the interface of MedlinePlus. As shown information organization. They agreed that information was in the table, participants expressed mixed feelings. Some clearly listed and logically organized, which provided a com- thought that the interface was simple, clean, easy on the prehensive map to the content of the system. Only one eyes, interactive, and visually pleasing, while others com- participant thought that there were not enough subgroups. mented that the interface was plain, outdated, cluttered, and lacked pictures. When evaluating navigations in the sys- Representations and evaluations of the interface compo- tem, participants overall had a positive feeling, agreeing nent. Participants represented the interface of MedlinePlus that the interface was streamlined, easy to navigate, and in relation to three aspects: interface elements, functions, and with decent flexibility. When evaluating the search function, results, as shown in Table 7a. however, they were more critical, pointing out that the sys- After using MedlinePlus for 5 minutes, participants rep- tem had limited searchability and there was no advanced resented mostly general interface elements, such as about search. us, disclaimer, logo, banner, contact us, links, questions and answers, menus, and tabs. Some of them mentioned a few Pragmatic knowledge of using MedlinePlus. As has been elements specific to MedlinePlus, such as purple, suggested introduced in the previous overview section, participants pro- articles, related illness, and hospital and doctor listings.When posed three distinct strategies to tackle a hypothetical task representing functions, participants emphasized the search using MedlinePlus: a general search strategy (strategy A), a function, particularly the general site-wide search (27 out browsing strategy (strategy C), and a combination of search- the 39 concepts), but they also recognized that they could ing and browsing (strategy B). Table 8 shows the proposed search for doctors or hospitals by state and search for different use of the strategies. As shown in the table, 24 (63.2%) par- medications. Several participants mentioned browsing as an ticipants planned to use either general search strategy A or option for looking for information in MedlinePlus. Two func- browsing strategy C to solve the hypothetical task. Fourteen tions that were not available in the system, symptom finder (36.8%) planned to use multiple strategies in combination. and pill identifier, were listed as well. It might reflect partic- Concerning the use of each individual strategy, 28 (73.7%) ipants’ need for the functions, as indicated by a comment in participants planned to use the general search strategy A. the semistructured interviews: Twenty-four (63.2%) planned to use browsing strategy C. ..., but they could have symptom checker on the homepage, Only one (2.6%) proposed to use strategy B, the strategy have instructions on if you want to find this information, go that was more aligned to the information architecture of here and if you want to find this information, go there. MedlinePlus.

2214 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—November 2010 DOI: 10.1002/asi TABLE 7b. Participants’ evaluations and emotions concerning the interface of MedlinePlus.

Look & Feel (35) – Clear layout – Old fashioned, A little outdated – Clean lines on Website – Not extremely visually appealing – Simple – Lack of pictures, Need some design improvement such – Plain as addition of images, Few pictures – Welcoming – Needs new Website banner – Pretty, Colorful – Word-intensive, Text-heavy – Visually pleasing – Sometimes too much information on a single page, – Catchy title – Many words on each page – No advertising! – Some pages overwhelming with choices of topics – Interactive – No clutter – Easy on eyes Navigation (14) – Streamlined – Decent ﬂexibility – Easy to navigate – Like the links to other pages – Ability to jump topics with ease – The variety of links on each page is helpful. – Easy to follow from link to link – Helpful links – Easy interface Search (3) – Lacking search functions – No advanced search – Limited searchability

When similar expressions were used to describe one concept, only one expression is listed in the table.

TABLE 8. Strategies that participants proposed to solve a hypothetical Measuring mental models is challenging. In this study task. we employed a new method, concept listing, to elicit par- Strategy No. of participants ticipants’ mental model of MedlinePlus. In the concept listing, participants were given an explicit instruction to a A 14 (36.9%) list concepts concerning MedlinePlus in the sequence that Cc 10 (26.3%) A + C 13 (34.2%) the concepts appeared in their mind. The method is a variation A + Bb + C 1 (2.6%) of word association tests, a method that has been widely used in the field of psychology to understand people’s knowledge aStrategy A: Search → read results. structures and associative cognitive processes (Cramer, 1968; b → → → Strategy B: Search top result health topic page review content Goldsmith, Johnson, &Acton, 1991; Pejtersen, 1991). Wang, and links. cStrategy C: Browse: Homepage → Health issues → health topic Bales, Reiger, and Zhang (2004) has used the concept listing page → review content and links. method to explore the development of students’knowledge of a subject domain over a semester. However, few studies of mental models adopted the method as a means to elicit people’s mental models of interactive information systems. Discussion and Conclusion The result of the study suggests that the concept listing Participants in the study were a group of homogeneous method was effective in representing mental models. Thirty- undergraduate students. They were able to articulate an eight participants listed, in total, 829 concepts. The concepts initial model of information-rich Web spaces. The initial described MedlinePlus from different aspects and allowed model had three distinct dimensions. The first was a rep- elements and dimensions of mental models to emerge natu- resentational dimension, in which participants represented rally. The method was also very efficient. It does not need various elements of information-rich Web spaces, such as interference on the researcher’s part in data collection and systems’ audience, subject and format of the information, only required 5 minutes from the participants. Five minutes tabs, links, and homepage. The representations clustered is reasonable because long pauses were observed toward the into four components: system, content, information orga- end of the session. The method, however, has a limited power nization, and interface. Simultaneously, participants formed in eliciting procedural knowledge. In the study, participants’ evaluations of or emotions about each of the components, procedural knowledge of using MedlinePlus was elicited by pointing out that information-rich Web spaces overall were asking them to describe steps that they planned to take to convenient and accessible. They contained a large amount solve a hypothetical task using the system. of information and the information was rigidly organized. The analysis of the concepts suggested that participants Their interfaces were easy to use, but could also be clut- represented MedlinePlus in relation to four components: tered and distracting. These evaluations/emotions constituted MedlinePlus as an integrated system, the content, the infor- the evaluation/emotion dimension. The third dimension was mation organization, and the interface of the system. At informed by the mental model theory and consisted of a set of the same time, they evaluated or expressed emotions about practical strategies or approaches for using the spaces, such each of the four components. They also proposed differ- as search, browse, and glance for graphs and headings. ent strategies to solve a task using the system. As a result,

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—November 2010 2215 DOI: 10.1002/asi participants’ mental model of MedlinePlus demonstrated of the interface, some thought it simple, clean, and welcom- a similar three-dimension and four-component structure to ing, while others thought it plain, outdated, and cluttered. their model of information-rich Web spaces. This similarity For the navigation, participants recognized that the interface suggests that the development of participants’ mental model was streamlined and easy to navigate. For the search func- of MedlinePlus, an instance of information-rich Web space, tion, participants were more critical, pointing out a lack of might be scaffolded by the structure of their mental model of advanced search. information-rich Web spaces in general. In terms of the procedural knowledge of using the system, Despite the structural similarity, participants’ mental the majority of the participants planned to use search, browse, model of MedlinePlus was more elaborated than the ini- or the combination of the two to solve a hypothetical task. tial model. When representing MedlinePlus as an integrated Only one participant proposed a strategy more aligned to the system, participants noticed the structure of the system, point- capacity of the system. ing out that the system was database-driven. It had its own Participants’ multidimensional and multifaceted mental information, but also linked to lots of outside information. model of MedlinePlus suggests that when people start to use They noticed agencies, such NIH and Centers for Disease a new system, they do not focus on only one or two aspects Control (CDC), involved in the system. They observed gen- of the system; rather, they represent and evaluate the sys- eral behavior of the system, such as pop-up windows. They tem from many different aspects, trying to decide what they thought about possible audiences for and potential usage of can do with the system, whether the system fits them, what the system. They also recalled two similar sites, WebMD and types information are available and in which formats, how PubMed, to help them make sense of the system. Simultane- information is organized, and what functions are provided. ously, they evaluated MedlinePlus as an integrated system in Based on the representations and evaluations, they formulate terms of its usefulness, usability, and some other attributes. perceptions about how to solve tasks using the system. They generally agreed that MedlinePlus was a data-rich and Although being multidimensional and multifaceted, par- search-oriented system. It was useful and user-friendly. ticipants’mental model of MedlinePlus after a brief exposure The content component was the most emphasized com- to the system was rather superficial. They tended to focus on ponent in participants’ mental model of MedlinePlus. This elements at the surface of the interface and easily observ- is not surprising, considering the fact that content often able information, such as the subject, type, and format of the matters the most to end users. In the study, participants content, and system behavior such as pop-up windows. Their represented general subject areas, such as medicine, symp- evaluations focused more on the easily observable aspects, toms, and treatments, and specific topics such as diabetes, such as usability, navigation, and the look and feel of the high blood pressure, and vitamin A. They also pointed out system. Few participants mentioned the underlying work- basic information types in the system, such as dictionary, ing mechanisms, such as how their search engine processes encyclopedia, news, and directories; general formats, such search queries and how MedlinePlus links all the content as image and PDF; and different ways in which the infor- together in an integrated manner. In terms of the procedural mation was presented, such as overviews and summaries. knowledge, only one participant identified a search strategy Similarly, participants quickly developed opinions about the more specific to the system. content. They were impressed by the quantity of information The construction of mental models is a result of a con- in the system and made judgments about its comprehensive- structive interaction between knowledge in the head and ness, currency, depth, as well as quality and utility. They also knowledge in the world (e.g., Jonassen & Henning, 1999). started evaluating some specific sections of content. The impact of the knowledge in the head on mental model is Information organization is important for a hyperlink- reflected by three observations in the study. First, the structure based Web system and is of particular interest to researchers and composition of participants’ mental model of Medline- in Information and Library Science (ILS) (Morville & Plus inherited that of their initial model of information-rich Rosenfeld, 2006). In the study, participants paid the least Web spaces. Second, some participants employed more attention to this component. Nevertheless, they were able to familiar systems, PubMed and WebMD, in their knowledge represent basic information organization schemas, such as base to assist them in understanding MedlinePlus. Metaphors alphabetical, hierarchical, by body systems, and by demo- were long considered as a tool that could facilitate learning graphic characteristics. At the same time, they formed a (Carroll & Olson, 1987; Staggers & Norcio, 1993). Third, positive feeling about this component, pointing out that participants incorporated two functions, “symptom finder” information in MedlinePlus was clearly listed and logically and “pill identifier,” in their current knowledge base but not organized. available in MedlinePlus into their mental model of the sys- In representing the interface of MedlinePlus, participants tem. The impact of the system (knowledge in the world) is mainly focused on the interface elements and functions. The reflected by the observation that although the structure of most-listed elements appeared at the surface of the sys- the mental model of MedlinePlus was similar to their ini- tem, such as contact us, disclaimer, tabs, and logo. The tial model of information-rich Web spaces, the substance most-emphasized functions were search and browse. Corre- was more MedlinePlus-specific, encompassing MedlinePlus- spondingly, their evaluations of the interface focused on the specific elements and functions and evaluations of various look and feel, navigation, and search. For the look and feel aspects of the system.

2216 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—November 2010 DOI: 10.1002/asi The research results have implications for system design. Borgman, C.L. (1986). The users mental model of an information-retrieval First, it is apparent that users perceived information-rich Web system—an experiment on a prototype online catalog. International spaces in relation to four components: system, content, infor- Journal of Man-Machine Studies, 24(1), 47–64. Brandt, D.S., & Uden, L. (2003). Insight into mental models of novice mation organization, and interface. To help users build correct Internet searchers. Communications of ACM, 46(7), 133–136. mental models of an information-rich Web space, design- Bruce, H. (1999). Perceptions of the Internet: What people think when they ers need to consider how to convey the system’s conceptual search the Internet for information. Internet Research, 9(3), 187–199. model in a way compatible with the ways in which users Burk, K.E., Martin, M.T., Reilly, C.A., & Kuperman, G.J. (2003). An perceive the system. Furthermore, designers could consider online consumer health information resource: 3-year usage summary. In American Medical Informatics Association 2003 Symposium Pro- how to present other less visible but important aspects of ceedings (p. 800). Bethesda, MD: AMIA. Retreived July 7, 2010, from the system to users. Second, in their brief encounter with http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1480148/pdf/amia2003_ MedlinePlus, participants tried to learn the system by pay- 0800.pdf ing attention to the structure, audience and potential usage Campagnoni, F., & Ehrlich, K. (1989). Information retrieval using a of the system, various types and formats of information, and hypertext-based help system.ACM Transactions on Information Systems, 7(3), 271–291. elements and functions at the surface of its interface. Partic- Carroll, J.M., & Olson, J.R. (1987). Mental models in human-computer ipants’ attention at this time suggests that system designers interaction: Research issues about what the user of software knows. Com- should convey such information in a conspicuous manner to mittee on Human Factors, Commission on Behavioral and Social Sciences help users develop a model of the system. Meanwhile, par- and Education, National Research Council. Washington, DC: National ticipants made quick judgments about the usefulness of the Academy Press. Chang, S.N. (2007). Externalizing students’ mental models through concept system, the quantity, quality, and utility of the content, and maps. Educational Research, 41(3): 107–112. the look and feel and navigation of the interface. How to Chen, C., & Czerwinski, M. (1997). Spatial ability and visual navigation: design interface cues to foster users to develop good impres- An empirical study. New Reviews of Hypermedia and Multimedia, 3(1), sions of these aspects of the system will also be worthwhile 67–89. to consider. Cool, C., Park, S., Belkin, N.J., Koenemann, J., & Ng, K.B. (1996). Information seeking behavior in new searching environments. Proceed- As with other studies, this study has limitations. Learning ings of the Second International Conference on Conceptions of Library a system is a developmental process, as is the construction of and Information Science (pp. 403–416). Copenhagen: Royal School of mental models of a system. This study only addresses peo- Librarianship. ple’s perceptions and representations of MedlinePlus after Cramer, P. (1968). Word association. New York: Academic Press. they used the system for 5 minutes. It will be worthwhile to deKleer, J., & Brown, J.S. (1983). Assumptions and ambiguities in mecha- nistic mental models. In G. Gentner &A.L. Stevens (Eds.), Mental models investigate whether users will switch their attention to some (pp. 155–190). Hillsdale, NJ: Lawrence Erlbaum Associates. other aspects of the system and whether there will be new Dimitroff, A. (1992). Mental models theory and search outcome in a bib- aspects from which they perceive and evaluate the system, liographic retrieval system. Library and Information Science Research, when their experience with the system increases. In the study, 14(2), 141–156. participants were asked to freely explore the system as they diSessa, A. (1986). Models of computation. In D.A. Norman & S.W. Draper (Eds.), User-centered system design: New perspectives in human- normally do. Although in some cases people explore a new computer interaction. Hillsdale, NJ: Lawrence Erlbaum Associates. system to learn about the system, many come to a system with Eckstrom, R.B., French, J.W., Harman, H.H., & Derman, D. (1976). Kit a speciﬁc task. So in future studies it will be worthwhile to of factor-referenced cognitive tests. Princeton, NJ: Educational Testing study people’s mental models of a system after they perform Service. a speciﬁc task. This study looks at people’s mental models Efthimiadis, E.N., & Hendry, D.G. (2005). Search engines and how students think they work. In Proceedings of the ACM Conference on Research of MedlinePlus from a collective perspective. Future studies and Development in Information Retrieval (SIGIR’05) (pp. 595–596). should look into individual users’mental models and explore New York: ACM Press. whether there are different patterns of models. Farris, J.S., Jones, K.S., & Elgin, P.D. (2002). Users’ schemata of hypermedia: What is so spatial about a Website? Interacting with Computers, 14(5), 487–502. Fein, R.M., Olson, G.M., & Olson, J.S. (1993). A mental model can help Acknowledgments with learning to operate a complex device. INTERACT ’93 and CHI I would like to thank my dissertation advisor, Barbara ’93 Conference Companion on Human Factors in Computing Systems (pp. 157–158). New York: ACM Press. Wildemuth, as well as my committee members, Gary Fox, S. (2005). Pew Internet & American Life project: Health information Marchionini, Paul Solomon, Diane Kelly, Javed Mostafa, and Online. Retrieved January 15, 2010, from http://www.pewInternet.org/≈/ Peiling Wang. I would also like to thank Andrew Dillon and media//Files/Reports/2005/PIP_Healthtopics_May05.pdf.pdf anonymous reviewers for their insightful comments on earlier Gentner, D., & Stevens,A.L. (1983). Mental models. Hillsdale, NJ: Lawrence drafts of this article. Erlbaum Associates. Goldsmith, T.E., Johnson, P.J., & Acton, W.H. (1991). Assessing structural knowledge. Journal of Educational Psychology, 83(1), 88–96. References Gott, S.P., Bennett, W., & Gillet, A. (1986). Models of technical competence for intelligent tutoring systems. Journal of Computer Based Instruction, Billinghurst, M., & Weghorst, S. (1995). The use of sketch maps to measure 13(2), 43–46. cognitive maps of virtual environments. In Proceedings of IEEE 1995 Vir- Graham, J., Zheng, L.Y., & Gonzalez, C. (2006). A cognitive approach to tual Reality Annual International Symposium (pp. 40–47). Washington, game usability and design: Mental model development in novice real-time DC: IEEE Press. strategy gamers. CyberPsychology & Behavior, 9(3), 361–366.

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—November 2010 2217 DOI: 10.1002/asi Halasz, F.G., & Moran, T.P. (1983). Mental models and problem-solving in Rode, J., Rosson, M.B., & Perez-Quifiones, M. (2004). End users’ men- using a calculator. In Proceedings of CHI’83 Human Factors in Computing tal models of concepts critical to Web application development. In Systems (pp. 212–216). New York: ACM Press. Proceedings of the 2004 IEEE Symposium on Visual Languages and Hanisch, K.A., Kramer, A.F., & Hulin, C.L. (1991). Cognitive represen- Human Centric Computing (pp. 215–222). Washington, DC: IEEE tations, control, and understanding of complex systems: A field study Press. focusing on components of users’ mental models and expert/novice Roger, Y. (1992). Introduction. In Y. Rogers, A. Rutherford, & P.A. Bibby differences. Ergonomics, 34(8), 1129–1145. (Eds.), Models in the mind: Theory, perspective and application. London: Johnson-Laird, P.N. (1983). Mental models. Cambridge, MA: Harvard Academic Press. University Press. Royer, J.M., Cisero, C.A., & Carlo, M.S. (1993). Techniques and procedures Jonassen, D.H., & Henning, P. (1999). Mental models: Knowledge in the for assessing cognitive skills. Review of Educational Research, 63(2), head and knowledge in the world. Educational Technology, 39(3), 37–42. 201–243. Katzeff, C. (1990). System demands on mental models for a fulltext database. Rutherford, A., & Wilson, J.R. (1992). Searching for mental models in International Journal of Man-Machine Studies, 32(5), 483–509. human-machine systems. In Y. Rogers, A. Rutherford, & P.A. Bibby Kerr, S.T. (1990). Wayfinding in an electronic database: The relative impor- (Eds.), Models in the mind: theory, perspective, & application (pp. 195– tance of navigational cues VS. mental models. Information Processing & 223). San Diego, CA: Academic Press. Management, 26(4), 511–523. Sasse, M.A. (1991). How to t(r)ap users’ mental models. In M.J. Tauber & Kieras, D.E., & Bovair, S. (1984). The role of a mental model in learning to D. Ackermann (Eds.), Mental models and human-computer interaction 2 operate a device. Cognitive Science 8(3): 355–273. (pp. 59–79). Amsterdam: Elsevier Science. Krippendorff, K. (2004). Content analysis: An introduction to its methodol- Savage-Knepshield, P.A.(2001). Mental models: Issues in construction, con- ogy. Thousand Oaks, CA: Sage Publication. gruency, and cognition. Unpublished doctoral dissertation, Rutgers, State Kudikyala, U.K., & Vaughn, R.B. (2005). Software requirement understand- University of New Jersey, New Brunswick. ing using Pathfinder networks: Discovering and evaluating mental models. Shneiderman, B. (1997). Designing information-abundant Web sites: Issues The Journal of Systems and Software, 74(1), 101–108. and recommendations. International Journal of Human-Computer Stud- Langan-Fox, J., Platania-Phung, C., & Waycott, J. (2006). Effects of advance ies, 47, 5–29. organizers, mental models and abilities on task and recall performance Sillence, E., Briggs, P., Harris, P.R., & Fishwick, L. (2007). How do patients using a mobile phone network. Applied Cognitive Psychology, 20(9), evaluate and make use of online health information? Social Science & 1143–1165. Medicine, 64(9), 1853–1862. Makri, S., Blandford, A., Gow, J., Rimmer, J., Warwick, C., & Buchanan, G. Slone, D.J. (2002). The influence of mental models and goals on search (2007). A library or just another information resource? A case study patterns during Web interaction. Journal of the American Society for of users’ mental models of traditional and digital libraries. Journal of Information Science and Technology, 53(13), 1152–1169. the American Society for Information Science and Technology, 58(3), Spink, A., & Cole, C. (2006). Human information behavior: Integrating 433–445. diverse approaches and information use. Journal of the American Society Marill, J.L., Miller, N., & Kitendaugh, P. (2006). The MedlinePlus public for Information Science and Technology, 57(1), 26–35. user interface: Studies of design challenges and opportunities. Journal of Staggers, N., & Norcio, A.F. (1993). Mental models—Concepts for human- Medical Library Association, 94(1), 30–40. computer interaction research. International Journal of Man-Machine Mayer, R.E. (2002). Multimedia learning. NewYork: Cambridge University Studies, 38(4), 587–605. Press. Thatcher, A., & Greyling, M. (1998). Mental models of the Internet. Muramatsu, J., & Pratt, W. (2001). Transparent queries: Investigating users’ International Journal of Industrial Ergonomics, 22(4–5), 299–305. mental models of search engines. In Proceedings of the 24th International Wang, P., Bales, S., Reiger, J., & Zhang,Y.(2004). Survey of learners’knowl- ACM Conference on Research and Development in Information Retrieval edge structures: Rationales, methods and instruments. In Proceedings of (SIGIR 2001) (pp. 217–224). New York: ACM Press. the 67th Annual Meeting of the ASIS&T (pp. 218–228). Medford, NJ: Morville, P., & Rosenfeld, L. (2006). Information architecture for the World Information Today. Wide Web: Designing large-scale Web sites. Sebastopol, CA: O’Reilly Wilson, J.R., & Rutherford, A. (1989). Mental models—Theory and appli- Media. cation in human factors. Human Factors, 31(6), 617–634. Norman, D.A. (1983). Some observations on mental models. In D. Gentner & Young, I. (2008). Mental models: Aligning design strategy with human A.L. Stevens (Eds.), Mental models (pp. 7–14). Hillsdale, NJ: Lawrence behavior. Brooklyn, NY: Rosenfeld Media. Erlbaum. Zhang, X. (1997). A study of the effects of user characteristics on mental Otter, M., & Johnson, H. (2000). Lost in hyperspace: Metrics and mental models of information retrieval systems. Unpublished doctoral disserta- models. Interacting With Computers, 13(1), 1–40. tion, University of Toronto, Ontario, Canada. Papastergiou, M. (2005). Students’ mental models of the Internet and Zhang, Y. (2008a). The influence of mental models on undergraduate their didactical exploitation in informatics education. Education and students’ searching behavior on the Web. Information Processing & Information Technology, 10(4), 341–360. Management, 44(3), 1330–1345. Pejtersen, A.M. (1991). Interfaces based on associative semantics for Zhang, Y. (2008b). Undergraduate students’ mental models of the Web as browsing in information retrieval (Tech Rep.). Roskilde, Denmark: Risø an information retrieval system. Journal of the American Society for Laboratory. Information Science and Technology, 59(13), 2087–2098.

2218 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—November 2010 DOI: 10.1002/asi