Running head: USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
The Influence of User Characteristics on the
Findability of Information on Health-Related Websites
Carlijn Raijmakers
ANR u431111, SNR 2001291
Master’s Thesis
Department Communication and Information Sciences
Specialization Business Communication and Digital Media
Faculty of Humanities
Tilburg University, Tilburg
Supervisor: dr. K.E. Bálint
Second reader: prof. dr. E.J. Krahmer
Supervisors IKNL: drs. M. van Eenbergen and T. Kolman
January 2018
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Table of Contents
Abstract ...... 4 Introduction ...... 5 Theoretical Framework ...... 8 Cancer Patients and Online Health Information ...... 9 Findability of Information ...... 10 Attitude towards Online Health Information ...... 13 Expertise in Searching for Online Health Information ...... 15 Illness Status ...... 16 Background of the Current Study ...... 16 Method ...... 18 Participants ...... 18 Materials ...... 19 Data Collection Methods ...... 19 Procedure ...... 20 Measures ...... 21 Findability of information...... 21 Attitude towards online health information...... 22 Expertise in searching for online health information...... 24 Illness status ...... 24 Method of Analysis ...... 24 Codebook...... 25 Coding procedure...... 27 Results ...... 28 Findability ...... 28 Attitude and Findability ...... 31 Performance...... 31 Perception...... 32 Expertise ...... 33 Performance...... 33 Perception...... 33 Illness Status ...... 33 Performance...... 33
2
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Perception...... 34 Discussion and Conclusion ...... 38 Limitations and Implications for Further Research ...... 41 Theoretical and Practical Implications ...... 43 Practical Implications for IKNL Specific ...... 44 References ...... 45 Appendices ...... 51
3
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Abstract
Sixty-three percent of the cancer patients search for information about cancer on the internet
(Castleton et al., 2010). Finding the desired information can reduce anxiety and uncertainty for patients (Leydon et al., 2000). This study was conducted to test whether cancer patients are able to find the information they want on the internet. More specifically, this study tested the findability of information on http://www.kanker.nl/verwijsgids. Moreover, it was tested if the user characteristic attitude towards online health information, expertise in searching online health information, and illness status are determinants for findability of information. Results showed that all three user characteristics are not significant determinants for findability of information. The performance metrics showed a good findability of information on the website. However, the frequency of negative statements in comparison to neutral and positive statements showed a negative perception of findability. To enhance the perception of findability, the issues are analysed and practical implications for IKNL are given.
Keywords: attitudes, cancer, patients, eHealth, illness status, expertise, findability, health communication, usability
4
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Introduction
Health information is one of the most frequently sought topics on the world wide web
(McMullan, 2006). Many definitions of online health information exist, but for this study, we assume online health information can be defined as the public using the internet to access health and lifestyle information, services and support (Wyatt, & Liu, 2002). Four-and-a-half percent of all searches performed on the internet are for health-related information (Morahan-
Martin, 2004). This percentage accounts for more than 12.5 million health-related computer searches each day (Eysenbach, 2003). Patients and physicians use health-related websites as an important information source, which can in turn lead to great benefits (Moreno, Morales del Castillo, Porcel, & Herrera-Viedema, 2010). More specifically, finding the desired information can reduce anxiety and uncertainty for patients (Leydon et al., 2000). However, this is not self-evident due to the rapidly growing online environment, which at the same time lacks publishing control and misses the guarantee of a website being highly qualitative
(Grigoroudis, Litos, Moustakis, Politis, & Tsironis, 2008).
Due to the large need for and importance of health information websites, their optimization is increasingly receiving greater attention in research (i.e., Kim, & Chang,
(2007)). Usability testing is a commonly used tool to optimize digital products. As stated by
Grigoroudis et al. (2008), assessment of a website can positively contribute to the development of a website, to serve the user’s needs, and to maximize the overlap between the website and users’ expectations. Moreover, it ensures that search goals can be accomplished with minimal cognitive effort (Jaspers, Steen, Bos, & Geenen, 2004).
Several studies have focused on usability, effectiveness and efficiency of websites.
One of these studies shows that in general, reasons for users to give up their information search can be divided into three parts: technical problems (e.g., time to download web pages),
5
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
the lack of information on the website, and bad usability of the website (Schenkman, &
Jönsson, 2000; Chevalier, & Kicka, 2006; Chevalier, Maury, & Fouquerau, 2014). When patients give up their information search, the benefits from finding information are missed, which makes a good findability important. However, the question of whether the findability is a result of usability, or is influenced by user characteristics, remains.
A group for which these benefits can certainly enhance their mental state, are cancer patients. For this particular group, communication and information are considered an important factor in developing a coping strategy to deal with cancer. Cancer can bring forth negative feelings like uncertainty and anxiety, yet may be reduced by providing information.
Which information is preferred, differs per patient, leading to differing effort patients are willing to put into searching (Leydon et al., 2000).
Multiple prior studies involving the role of online health information for cancer patients have been conducted. For instance, a cross-sectional study among a group of cancer patients addressed the need for supportive care and attitudes towards online health information. Moreover, the study included the role of sociodemographic and clinical variables alongside the quality of life. Results show that in general, cancer survivors have a positive attitude towards online health information. Moreover, the positive attitude towards online health information is associated with lower age, higher education, higher income and currently being under treatment. Finally, attitudes towards online health information is related to life quality (Jansen, Van Uden-Kraan, Van Zwieten, Witte, & Verdonck-de Leeuw, 2015).
However, much less is known about the relationship between the attitude towards online health information to usability or, specifically, to findability of information. Stated differently, issues with findability may not only stem from usability issues, but may be influenced by user characteristics. It is well established that attitude influences behavior, but, to our knowledge,
6
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
no prior studies relate attitude towards online health information to the findability of information.
Another study that investigated this specific group included 200 breast cancer patients.
Results reveal that 31% of the patients who used the internet to gain information about breast cancer experienced problems in finding the information they were looking for (Littlechild, &
Barr, 2013). Limited research has been done on the capabilities of cancer patients to search for online health information. However, research suggests that unfamiliarity with the internet can be a disadvantage when using the internet in comparison with people that are more knowledgeable about the internet and its usage (Potosky, 2006). Experience in using the internet, which in the current study is referred to as expertise, is one of various individual characteristics that could affect information search (Schenkman, & Jönsson, 2000; Chevalier,
& Kicka, 2006; Chevalier, Maury, & Fouquerau, 2014).
Most health-related websites are not specifically developed for patients. Patients and non-patients have a different illness status. Results of a study by Xiao, Sharman, Rao, and
Upadhyaya (2014) show that cancer patients tend to seek cancer information more intensively than healthy individuals (i.e., non-patients). The latter group has presumably much less (i.e., a lower frequency) experiences with websites about cancer than those who have suffered from cancer, and their attitude towards online health information may not be formed yet (i.e., negative or positive). These differences may influence findability of information as well.
Considering that user characteristics attitude towards online health information, the expertise with using the internet for online health information, and illness status are important aspects in looking for online health information, it is relevant to investigate whether these aspects determine the findability of the information to enhance findability of information for cancer patients.
7
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Extensive studies exist that include online health information, the credibility of information of cancer-related websites (e.g., Khazaal et al., 2009), the need for online information and the reasons to search for online information. However, some gaps in the literature must be acknowledged. More specifically, most studies focus on the features of the website regarding findability and do not take user characteristics into account. Therefore, this study is a first attempt to explore the relation between attitudes towards online health information, the expertise in searching for online health information, illness status and the findability of information. This leads to the central research question of this thesis: “To what extent is information findable on cancer information-related websites?”
To answer the research question, first, a theoretical framework is developed.
Subsequently, the data is gathered by using a survey and thinking aloud method. Patients and non-patients participated in the study. Eventually, the outcomes contribute to theory by providing information about the influence of user characteristics on findability and may bring about a shift in focus from the features of the website to the user characteristics. Moreover, practical implications will be given to enhance the findability of cancer information-related websites.
Theoretical Framework
First, the use of online health information of cancer patients is explained. Secondly, findability of information is explained from two different viewpoints: the website’s features and the user’s side. Thirdly, the (possible) role of user characteristics is considered. Attitude towards online health information, expertise in searching for online health information, and illness status are discussed in depth.
8
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Cancer Patients and Online Health Information
In this section, several aspects of a cancer patient’s use of online health information are explained. More specifically, it is explained how many cancer patients use online health information, which subject characteristics predict searching for online health information and the goals of cancer patients in searching for online health information (eHealth). In this study, the focus is placed on online health information, which is also known as eHealth.
Information and communication technologies have seen an enormous usage increase in recent years, just as online health information. This has led to evermore cancer patients finding their way into online health information. Meta-analysis as conducted by Eysenbach
(2003) has shown that approximately 39% of cancer patients are using the internet. In fact,
15% to 20% can be added to that number, accounting for indirect use of internet via family and friends. This percentage grew to 63% in 2010 (Castleton et al., 2010).
To retrieve more knowledge about the group of cancer patients that search for online health information, multiple studies have included user characteristics to see if those are related. For example, several studies showed a significant difference in age and searching for online health information about cancer. A questionnaire held for Australian oncology patients also showed a significant association with a younger age (Newnham et al., 2006).
Additionally, information seeking was significantly predicted by different variables. First of all, age, where more information seeking was found by people who were younger than 65 years old. Second of all, gender, where more information seeking was found by females.
Third of all, income, where more information seeking was found by people earning more than
US$ 25,000 a year (Mayer et al., 2007). These results were also seen in a more recent study, which found that a younger age, a higher education, and a higher income predict the frequency of searching for health information from the internet. (Xiao et al., 2014). In line
9
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
with these studies, Castleton et al. (2010) found that 315 out of 500 respondents (63%), who all were cancer patients, search for information about cancer on the internet. Internet usage differed per age, race, and education.
Among varying groups that search for online information, information needs can vary among different types of cancer (Nagler et al., 2010), and among different roles (e.g., patients versus relatives). During several stages of cancer, both patients and their relatives are involved in decision making for example, about medical care and palliative care, from which information about the latter is offered on http://www.kanker.nl/verwijsgids. Not only for this website, but in general the world wide web has become a tool that is used extensively to search for information and services (Eysenbach, Powell, & Kuss, 2002). The main goal for searching for online health information is finding the desired information, preferably with minimal cognitive effort (Jaspers, Steen, Bos, & Geenen, 2004). Reaching this goal has proved to have great benefits for cancer patients. Several studies have shown that cancer patients perceived better outcomes, better coping with their condition, having less uncertainty, having less anxiety, and having, to a lesser extent, depression (Leydon et al., 2000;
LaCoursiere, Knobf, & McCorkle, 2005).
In sum, the majority of cancer patients use the internet to search for health information with as main goal finding information. Achieving this goal can lead to great benefits for cancer patients and can improve the quality their lives during the illness.
Findability of Information
The previous section explained the importance and benefits of finding the desired information on the internet. This section continues this by elaborating on how this goal (i.e., the findability) can be achieved or hindered.
10
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
To achieve a good findability of information, evaluation of websites is important. This has been acknowledged among scholars. The world wide web has grown rapidly, and due to the lack of publishing control, the quality of websites cannot be guaranteed. A questionnaire that was completed by 500 cancer patients, showed that patient-friendly language (74%), professional source (73%), and a well-designed website (58%) are important aspects of cancer-related websites (Castleton et al., 2010). To test and to revise these aspects, evaluation studies are needed. As a result of the fast growing content on the web, many scholars aimed to assess the quality of the web (Grigoroudis et al., 2008).
Besides assessing the quality of content, usability testing has become increasingly important. As pointed out in the introduction, results showed that reasons for users in general to give up their information search can be divided into three parts: technical problems (e.g., time to download web pages), the lack of information on the website, and bad usability of the website (Schenkman, & Jönsson, 2000; Chevalier, & Kicka, 2006; Chevalier, Maury, &
Fouquerau, 2014).
Usability is a broad concept, which involves multiple aspects, such as design and content. Since not all aspects of usability are covered within the study, the term findability is used. The definition of findability can be split up into three parts: (a) the quality of being locatable or navigable; (b) the degree to which a particular object is easy to discover or locate;
(c) the degree to which a system or environment supports navigation and retrieval (Morville,
2005).
A study that focused on findability of information for non-patients, is conducted by
Simunich, Robins, and Kelly (2015). In the study, the scholars made use of several usability testing methods, including think-aloud and eye-tracking. The problems that occurred highlighted the importance of navigation design and visual design. When participants experienced problems regarding these topics, a few categories could be the cause of these
11
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
problems. The categories were lacking chunking, poor labelling, poor categorization, deeply buried content, and a lack of visual contrast among page elements.
As been noted by Dhyani, Keong Ng, and Bhowmick (2002) most established models evaluating websites are mainly objective, qualitative and focused on usability, effectiveness, and efficiency of websites. Within these models and studies, user perception is not taken into account. Due to the lack of scientific definition of the perception of findability, the term is defined for this study as all perceptions of the user about findability of information, and all aspects that are involved in finding the information (e.g., navigation).
A usability problem that can be perceived as hindering findability of information, and often seen as most frustrating, is becoming lost or disoriented when navigating through websites. This phenomenon is referred to as “lostness” (Otter, & Johnson, 2000). Lostness is an important subject since the main goal of websites is to help users find the information they are looking for. When a user is lost, this goal is not accomplished. As a result of a study by
Elm and Woods (1985), three forms of being lost were distinguished. The first problem refers to not knowing where to go next, the second problem is knowing where to go, but not knowing how to get there, and the last problem is not knowing where they are in the overall structure of the document. Statements that are in line with this distinction are “I do not know where to go now.” and “Where am I in this website?” By retrieving this information, the perceived findability of a website can be assessed and thereby issues can be located and revised to enhance the findability of information.
A study that enlightens problems with findability for cancer patients is conducted by
Littlechild and Barr (2013). A questionnaire that was completed by 200 breast cancer patients revealed that 31% of the patients who used the internet to gain information about breast cancer experienced problems in finding the information they were looking for. In this study, ethnicity affected the likelihood of experiencing problems. More specifically, non-white
12
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
patients (64%) experienced more problems than white patients (25%) (Littlechild, & Barr,
2013). These outcomes show that problems with findability of information can result from user characteristics, and thus do not necessarily result from a bad usability.
In sum, many studies focused on usability to improve websites. However, less attention has been paid to the perception of patients when it comes to findability of information. A study that focused on findability of information with breast cancer patients, revealed that there are user characteristics that are related to a bad perceived findability (i.e., the participants experienced problems with finding information) (Littlechild, & Barr, 2013).
This prior study shows the problem with findability lays not necessarily with the website and its features, but assigns the cause at least partially to user characteristics.
Attitude towards Online Health Information
To elaborate on the possibility of user characteristics having an influence on findability of information, multiple user characteristics are discussed in this section and the following sections. First, attitude towards online health information is discussed.
One user characteristic is attitude. More specifically, attitude towards online health information. There are multiple aspects that can lead to different attitudes towards online health information, wherein attitude is defined as feeling or opinion. Examples that could lead to positive attitudes due to online health information are: 91% perceive a growing confidence to ask questions to a physician, and 92% perceive a greater empowerment to make decisions about their health. Examples that could lead to a negative attitude should not be neglected either. 31% of the cancer patients feel that online health information is overwhelming, 76% find conflicting medical information regarding cancer, and obtaining information about what the right treatment is confused 27% (Eysenbach, 2003).
13
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
A cross-sectional study that was conducted with a group of cancer patients studied the need for supportive care, and attitudes towards online health information. Cancer survivors had a positive attitude towards online health information. Moreover, the positive attitude was associated with lower age, higher education, higher income and currently being under treatment (Jansen et al., 2015). This study did not relate the findings to findability of information.
Xiao et al. (2014) conducted a national cancer-related survey about the frequency of online health information search, the diversity of online health information usage, and the preference of the internet for initial search. Results showed that if a user’s trust of online health information is high, this person is likely to have (a) a higher frequency of using the internet for searching for health information, (b) a higher preference in searching for online health information, and (c) have a more search diversity.
As can been seen, multiple studies relate attitudes towards online health information to several aspects that include searching for online health information. However, no studies have included attitudes towards online health information, have also focused on the relationships between attitude towards online health information and usability, or in specific, to findability of information. It is well established that attitude influences behavior (e.g., Theory of planned behavior (Ajzen, 1985)). To optimize findability of information for cancer patients, it is important to study the relationship between attitude towards online health information and findability of information. Therefore, the first research question is:
RQ1: To what extent does a user’s attitude towards online health information influence the findability of information?
By studying this user characteristic, the focus may shift from improving the websites usability to enhancing the attitudes of users towards online health information.
14
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Expertise in Searching for Online Health Information
Besides attitude towards online health information, expertise in searching for online health information may also determine the findability of information. In this study, expertise is referred to as the skill or knowledge someone has regarding a subject, which in this case is searching for online health information (Cambridge Dictionary, n.d.).
It is suggested that familiarity with using the internet is advantageous when searching on the internet in comparison with people that are not experienced in using the internet
(Potosky, 2006). To test this experience, multiple studies used the Internet Knowledge measures (iKnow measures) developed by Potosky (2006). Experience in using the web, which in this study is referred to as expertise, is one of various individual characteristics that could have an effect on information search (Schenkman, & Jönsson, 2000; Chevalier, &
Kicka, 2006; Chevalier, Maury, & Fouquerau, 2014). Research found that a user's trust toward online health information is positively related to the frequency of using the internet for health information. (Xiao et al., 2014). This, in turn, can lead to a greater expertise with searching for online health information.
Although limited research has been done that specifies the expertise of cancer patients in searching for online health information, some issues that may arise from a lack of expertise have been found. For example, in a study by Littlechild and Barr (2013) that was conducted with breast cancer patients, 31% of the patients experienced trouble in finding information on the internet.
By studying the influence of a user’s expertise in searching for online health information on the findability of information, it can be distinguished whether the findability of the website is below par when it comes to usability, or that a lack of expertise causes issues with findability of information. Therefore, the second research question is:
15
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
RQ2: To what extent does a user’s expertise in searching for online health information influence the findability of information?
Illness Status
In this section, the last user characteristics that may determine findability of information is examined, namely illness status. For this study, illness status is defined as the distinction between being a cancer patient or relative (hereafter referred to as ‘patient’) or not begin a cancer patient (hereafter referred to as ‘non-patient’).
Results from a study from Xiao et al. (2014) showed that cancer patients tend to seek cancer information more intensively than healthy people (i.e., non-patients). Although we know that patients search for information more intensively than non-patients, we do not know if this also leads to finding information more or less easily because patients search intensively and for more detailed information, which can be harder to find.
By studying the influence of illness status on findability of information, we can gain knowledge about the differences and non-differences between both groups. By gaining this knowledge, it may become clear whether both groups can use the same websites or that their illness status brings forth other needs (i.e., a special website for cancer patients). Therefore, the third, and last research question of this study is:
RQ3: To what extent does a user’s illness status influence the findability of information?
Background of the Current Study
A verified website that provides information on palliative care for cancer patients is www.kanker.nl/verwijsgids, which is owned by The Netherlands Comprehensive Cancer
Organization (IKNL hereafter). IKNL is the quality institute for oncological and palliative research and practice. The objective of IKNL is to serve in the public interest by promoting
16
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
the fight against cancer, particularly by helping those suffering from cancer and by promoting palliative care (IKNL, 2010). For this purpose, IKNL has created several websites. For example, www.kanker.nl/verwijsgids. During the development of the website www.kanker.nl/verwijsgids, patients were asked in which way they retrieve information to understand what suits the patients best. After a few years of development, the website came into use. However, due to the influence of multiple parties, compromises had to be made, that may have led to the focus on the needs of the cancer patients becoming less of a priority. For that reason, IKNL is not convinced the website fully adheres to patients’ needs and asked students of Tilburg University to evaluate the findability of the information that is displayed on the website. As stated in the introduction, it is important to evaluate websites. Moreover, a study from KWF Kankerbestrijding, a Dutch non-profit organisation that facilitates cancer research, showed that 86% of people that are or were treated for cancer deals with complaints
(i.e., depression, anxieties, relationship problems, hair loss). Complaints that are considered to be outside the range of cancer itself, can be treated by people who are called palliative carers
(KWF, n.d.). Thus, besides treatment for the disease, patients and relatives may also search for information for palliative care, which makes evaluating this website important well.
Therefore, this website is used in this study. By doing this, not only the influence of attitude, expertise, and illness status on findability of information can be studied, but also implications for improvement of the website can be given.
In sum, this study focuses on findability of information on the websites of IKNL and the influence of three user characteristics on findability, namely, attitude towards online health information, expertise in searching for online health information, and illness status.
And, studies their effect on the findability on information (see Figure 1).
17
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Attitude towards online health information
Expertise in searching for Findability of information
online health information
Illness status
Figure 1. Conceptual model
Method
As can be seen in Figure 1, the independent variables of this study are attitude towards online health information, expertise in searching for online health information, and illness status. The dependent variable is findability of information. Data is gathered by conducting a think-aloud session and survey.
Participants
In total, 22 people (18 woman, 4 men) with a mean age of 31.59 (SD = 16.78 ) participated in this study. Out of 22, 6 were patients (2 men, 4 woman) and 16 were non- patient (2 men, 14 woman). The mean age differed per sample, where patients were generally older (M = 56.83, SD = 10.91) than non-patients (M = 22.13, SD = 2.00). Patients were recruited by employees of IKNL via their online research panel and were representative end users of the website, in this case cancer patients and relatives. Non-patients were recruited by the researchers from their network and were all students.
Four of them graduated middle level education (MBO), eleven of them from higher education (HBO), two of them have a university degree, one of them only completed high school, and four of them belonged to the category ‘other’.
18
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
From this group, one participant visited the website once, six participants visited the website a couple of times, and fifteen participants did not visit the website prior to this study.
Materials
For this study, the findability of information for the website http://www.kanker.nl/verwijsgids was examined. The website’s main goal is to provide
(former) cancer patients and relatives with information, and in this case specifically helping patients to find palliative care matching their problems and geographical region. The goal of providing cancer patients with information matching their problem, and this study’s goal of measuring the findability of information, complement each other. Therefore, the website http://www.kanker.nl/verwijsgids was used as a stimulus for this study.
Data Collection Methods
For this study, two methods were used: a survey and a think-aloud session. The survey was cross-sectional and captured various user characteristics at one point in time. An advantage of using a survey is that respondents can answer large numbers of questions in a relatively short amount of time (Treadwell, 2017). For this study, this is most beneficial because the think-aloud session is already time-consuming. Moreover, the survey enables the possibility to use well-established constructs for measuring the user characteristics that are at hand in the current study.
According to Nielsen (1993), the think-aloud method may be the single most valuable usability engineering method. According to Krahmer and Ummelen (2004), this method has been used for three types of goals. One of the three goals is to assess issues in new documents or applications with the revision and improvement as a goal, for example, by usability testing.
The latter is consistent with the goal of IKNL and this study. During think-aloud sessions, participants use the website for a given set of tasks while they think-aloud (i.e., verbalize their thoughts). By verbalizing their thoughts, it can be determined what users are doing with the
19
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
interface, and why they are doing it. The observer is enabled to gain knowledge about the interface elements that cause misunderstandings and need to be revised (Nielsen, 1993).
Moreover, other professionals in this field perceive the think-aloud method (i.e., an in-person study) as the best way to facilitate identifying usability issues (Tullis, & Albert, 2013).
Procedure
All participants contributed to this research voluntarily. IKNL contacted patients and relatives. For patients and relatives, the full research procedure took place at their own homes or at the office of IKNL. For the non-patients, the study could also take place at Tilburg
University.
First, a short introduction took place and the participants signed a consent form (see
Appendix C). Subsequently, participants filled out the first part of the questionnaire. Next, the researcher explained the thinking aloud method. During this explanation, the researcher searched for a specific word in a dictionary while thinking aloud, to illustrate the explanation given, as advised by Nielsen (1993). Then, the participants practiced with thinking aloud themselves by using the Tower of Hanoi. The participants continued with the exercise until they’d convinced the researcher of the understanding of the method. Thereafter, participants received five tasks (see Appendix B). The tasks were based on the requirements as given by
Nielsen (Nielsen Norman Group, 2014). These requirements include that the task is representative with how people use the system when they use it on their own, the task should encourage users to interact with the interface, and the task should not give away the answer.
Due to ethical reasons (i.e., the heaviness of the topic) the researcher wrote the scenarios as if
‘Person A’ needed the answer to the given question, and not the participant his/herself. IKNL and the researcher chose the order of the scenarios to be fixed. The first and last scenarios were seen as relatively easy and not emotionally loaded in order to comfort the participants at the start and the end of the session.
20
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
After reading the handout, participants could ask questions if aspects were unclear.
Where needed, these were explained. After a clear understanding of the handout, the screen- and voice recording started and note-taking began. IceCream Screen recorder recorded the sound and clicking path of participants. During the tasks, the researcher only answered the questions from the participants if the questions were so urgent that the ignorance of them would stop the process, it was tried to give a counter-question like “What will happen if you do it?” (Nielsen, 1993). When participants stopped verbalizing their thoughts, the experimenter interrupted with questions such as “What are you thinking now?”. Between tasks, urgent questions could be asked. This procedure was repeated for five tasks. After completion of the tasks, the researcher asked questions elaborating on the verbalized thoughts of the participant. Next, the participant filled out the second part of the questionnaire. The non-patients were exposed to the procedure of Fabiënne Frijters too and thus participated in a second thinking aloud session for another website. The presentation order of the websites was randomized.
At the end of the session, the researcher thanked the participants for their help. A more elaborate version of the procedure can be found in Appendix D.
Measures
Findability of information. To measure the findability of information, a few measures are based on the most relevant parts of usability as explained by Quesenbery (2004) and the most relevant quantifiable usability measurements as proposed by Nielsen (1993), namely efficiency and effectiveness. These measurements measure the performance aspect of findability.
Efficiency is operationalized as the time needed for finding the answer to the task, measured in seconds. It is acknowledged that the method itself can slow task performance, but this means task performance is slowed for every participant within this study, and findings
21
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
can be compared within the study itself. In general, participants needed 150.09 (SD = 28.48) seconds to complete the task.
For effectiveness two components are distinguished: (a) the correctness of the answer is measured as correct or incorrect, and (b) the error rate is measured in the number of clicks that are divergent from the ideal path (i.e., the path with the least clicks needed to achieve the task). In general, participants needed 4.44 (SD = 1.46) extra clicks to complete the task and had a mean correctness of 0.84 (SD = 0.04), which represents correct answers for 84% of the given tasks.
To measure perceived findability, participants were asked to verbalize their thoughts while completing tasks. These thoughts were coded and categorized. In general, statements were categorized on the basis of their sentiment (i.e., negative, neutral or positive statements) and the type of issue (e.g., navigation problem). Statements were counted within the categories (see Table 1 and 2) for a quasi-statistical analysis. On average, participants verbalized 4.14 (SD = 2.30) negative statements, 2.45 (SD = 1.74) positive statements, and
2.73 (2.07) neutral statements.
To test for the intercoder reliability, 14% of the conversations were coded by an independent coder. Cohen’s Kappa was run to determine if there was an agreement between the two coders. There was a moderate agreement between the two coders, κ = .682 (95% CI,
.506 to .825). While calculating the Cohen’s Kappa, 13 variables of the 26 were accompanied by a warning (see Appendix F for the overall table). Although these variables represented a full agreement, no significance level could be presented. Only two subcategories (Stuck in loops, Other) had low Cohen’s Kappa values, κ < .60. These subcategories were thoroughly discussed and evaluated between coders. Eventually, full consensus was reached.
Attitude towards online health information. For this study, the e-Health Impact
Questionnaire was used, and additional questions that were added. The e-HIQ questionnaire
22
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
has been designed and developed to be used as an instrument to assess the effects of websites containing health information (Oxford University Innovation, n.d.).1 The eHIQ-NL consists of two independently administered and scored parts, that are divided into subscales. Part one includes 11 items that represent general attitudes towards using the internet to access health information, and consists of subscales ‘Attitudes towards online health information’, and
‘Attitudes towards online sharing of health experiences’. Whereas the first part measures the attitudes in general, the second part measures specific attitudes towards one health information related website. The second part includes 26 items that measure a person’s ease with using online information, and particularly emphasising a person’s openness to learning and gaining support from other people’s experiences. The second part is divided into three subscales that consist of ‘Trust and identification’, ‘Information and presentation’, and
‘Understanding and motivation’ (Oxford University Innovation, n.d.).
For this study, only the first scale (i.e., attitude towards online health information) from the eHIQ-NL was used and is measured with five items. Attitude is operationalized as a feeling or opinion about something, which in this case is online health information
(Cambridge Dictionary, n.d.). The items were measured on a 5-point Likert-scale. The response categories ranged from 1 (strongly disagree) to 5 (strongly agree). An example of an item is “Het internet is een betrouwbare bron om mij te helpen begrijpen wat een arts vertelt.”
(“The internet is an important source to help me understand what a doctor tells me.”) The scale had a mean score of 3.2 (SD = 0.53) and the scale’s reliability was marginally sufficient
(α = .55).
1 This questionnaire is translated into Dutch (i.e., eHIQ-NL) corresponding the COSMIN criteria (Mokkink et al., 2009). Currently, PhD-student Koen Neijenhuijs and professor doctor
Irma Verdonck-de Leeuw are validating the Dutch version of this questionnaire.
23
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Expertise in searching for online health information. Expertise is the level of knowledge or skill someone has (Cambridge Dictionary, n.d.). The 6 items were based on the
Internet Knowledge Measure (i.e., iKnow measure) by Potosky (2006) and adjusted to the context of this study (i.e., the search for online health information). The original Internet
Knowledge Measure measures internet knowledge in general. For this study, the items were focused on health information related websites. The response categories ranged from 1
(strongly disagree) to 5 (strongly agree). An example of an original item is “When using the
Internet, I quickly find information that I am looking for.”, which was adjusted to “When searching for health information, I quickly find information that I am looking for.”, and then translated to “Wanneer ik online zoek naar gezondheidsinformatie, vind ik snel wat ik zoek.”
(see Appendix A). The scale had a mean score of 3.55 (SD = 0.53) and the scale’s reliability was acceptable (α = .64).
Illness status. During the sampling period, patients and non-patients were sampled.
For certainty, illness status was measured with one question, where it was asked whether someone participated either as a patient or relative (N = 6) or as non-patient (N = 16).
Method of Analysis
To test whether attitude towards online health information (RQ1) and expertise in searching for online health information (RQ2) are related to the findability of the desired information, correlation analysis was conducted for the total sample, the patient, and non- patient sample. To test the differences between the two subsamples on findability (RQ3), multiple independent t-tests were conducted.
To test the general research question, the variable findability is split up into two parts: performance and perception. The perception part is coded and analysed. To test the findability, the perceptions were split up based on their sentiment (i.e., negative, neutral or
24
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
positive) and the type of issue (e.g., navigation problem). Issues have been split based on the impact they have had on the perceived findability. Additionally, for each category, and separately for every sample, the frequencies of given statements regarding findability are counted.
Codebook. The theory-driven part of the codebook is based on codebooks that were used for usability testing. The navigation problems are derived from Krahmer and Ummelen
(2004) and are supplemented with the categories of information processing by Eveland and
Dunwoody (2000). In Table 1, the theory-driven categories are displayed, together with an example from the transcriptions for one of the sentiments per category. Table 2 shows the data-driven categories. The creation of those categories is described in the next section.
25
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Table 1
Theory-driven categories Category Description Example
Certainty about Subjects see where they could go (click) N: “Okay, I get many results. I will action planning next or they see several possibilities but click on the first one.” have no clue about which one to select (-) Orientation Subjects (do not) understand where they N: “Oh, actually, I do not know are or can (not) interpret their current where to go.” location in the context of other locations on the website; asking rhetorical questions about the structure of the site; synthesizing the structure of site Expected result Subjects expect a certain result after P: “I am searching for contact with clicking a link, and this result does occur fellows. Let’s see. It’s not right here. (+) or this result does not occur (-). Ho! No, it’s not here neither.” Reasoning about Subjects start reasoning about why the N: “Maybe it is more like informal navigation logic interface makes them think they can find care. I will search for that. This is for certain information behind a link people who want to give informal care.” Interface Subjects verbalize positive words (+), or P: “How do I get this thing smaller? manipulation verbalize problems (-) about handling Oh over here, here is a minus sign.” certain objects in the interface (e.g.
dragging a pointer over a calendar in order to get to information about a certain year) Stuck in loops Subjects think they move on to a different P: “Oh.. I am in a loop here. That’s location on the website, but appear to not handy.” (repeatedly) return to where they came from Structure: Stating the name of links before or as they P: “I will search for contact with maintenance are traversed fellows. Contact. Patients. Breast cancer.” Structure: Expressing affection for the organization P: “Actually, I think it is organised evaluation of information or non-content features of neatly for a website.” site.
Note. N stands for non-patient and P for patient.
26
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Table 2
Data driven-categories Category Description Example Use of jargon The subject has trouble with understanding N: “What is an edema?” words that are used in the text Lack of The subject has trouble with completing N: “But this ‘haptotherapy’ is not information the task due to lack of knowledge (and completely clear to me. Not even about specialism information) about certain specialisms when I click on it, they say what they do but I don’t know..” Search topics The subject does not know where to search N: “Nope, this also does not exist. I next or what search words to use. This is do not really know what to do next. independent from orientation and Maybe I should type this again navigation logic. because I forgot is was about breast cancer. No, it is not here yet. I don’t know.” Preference for other The subject wants to continue the P: “I would go to Google.” sites search via Google or another website. Note. N stands for non-patient and P for patient.
Coding procedure. After finishing the transcription of the recordings, the researcher highlighted the relevant parts and split them up into statements. A statement is one or multiple sentences about one specific problem and are not intermittent by words about other categories. The coding procedure is represented in the following steps:
Step 1. If a statement reflected on a code that was already in the codebook, the researcher assigned that code to the statement. For all available codes, the researcher checked if those are applicable to the statement. The additional coding accounts for the sentiment of the statements if applicable with a (+) for positive statements, a (-) for negative statements, and a (=) for neutral statements.
27
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Step 2. If an uncoded statement involved relevant pieces after step one, the researcher noted down the statements and a new category arose in the data-driven part of the codebook.
If the statement did not directly belong to a certain code, the researcher set the code aside in a loose category, as in verbatim analysis (Rugg, & Petre, 2016).
Step 3. During the gist analysis, the researcher looked at the verbatim categories again to see if the categories are alternative phrasings of the same thing and do belong together. The researcher added a new category if needed.
Step 4. During the superordinate analysis, the researcher analysed the codings that were related but somehow were not clearly synonymous (Rugg, & Petre, 2016). The researcher added a new category if needed.
Step 5. After coding the absence or presence for each category for each statement, the transcriptions were analysed again, and frequencies were added. The transcriptions were coded by one researcher, namely the author of this thesis, Carlijn Raijmakers.
Results
First of all, this section shows results of the findability of information from the website. Second of all, it is explained whether attitude towards online health information, expertise in searching for online health information, and illness status are determinants for this findability.
Findability
To represent an overview of the problems that occurred during the tasks that influenced the findability, an issue analysis was made (see Figure 2). Since this analysis concerns problems, only negative statements were considered. Moreover, contrary to the tables that are provided for the frequency of statements, the given numbers in this analysis
28
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
represent unique problems. Subsequently, the problems were not rated for each participant, but for each task. Since there were five tasks for each participant, a problem could be coded five times per participant but not five times within one task executed by one participant.
(N = 22) Few users experiencing a problem Many users experiencing a (< 8) problem ( =/> 8) Small impact on Low severity Medium severity findability
Certainty about action planning (6) Interface manipulation (9) Orientation (6)
Large impact on Medium severity High severity findability
Stuck in loops (7) Expected result (26) Use of jargon (2) Structure: evaluation (14) Lack of information given about Not knowing which search specialism (4) topics to use (10) Verbalizing the preference for another website (6) Figure 2. Findability issues for total sample
The most frequently mentioned negative statements were about the category ‘expected result’ (36 times). Stated differently, participants experienced not finding a result which they expected to find on a certain page 36 times in total. This was followed by the category
‘Structure: evaluation’, which holds several statements about the organization of information and non-content features of the website. Most statements that were given within this category, had a large impact on the perceived findability. Another problem that occurred quite often (10 times) was that participants did not know which search words one could use to complete the task. Several problems occurred less often but did impact the perceived findability for some participants considerably. Multiple participants found the same loop for the third task, which
29
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
meant extra clicks were needed to complete the task. Multiple participants experienced problems with dealing with some elements in the interface, for example, filter options with boxes that could be ticked, or fill-in boxes which could be filtered on distance. In general, participants used filters 36.36% of the time, wherefrom 47.5% occurred in the last scenario, where the use of filters was obligatory.
The frequency of statements within each category is provided in Table 3. Existing categories without any statements, and thus with a frequency of zero, are omitted from the table (this holds for all similar tables that follow).
Table 3
Frequencies of statements Subcategory Negative statements Positive statements Neutral statements Certainty about action 6 6 15 planning Orientation 8 4 1 Expected result 36 17 * Reasoning about navigation 0 4 17 logic Interface manipulation 10 5 1 Stuck in loops 7 * * Structure: evaluation 15 16 3 Structure: maintenance * * 29 Use of jargon 2 * * Lack of information given about specialism 4 * * Search topics 10 * * Preference for other sites 6 * * Note. * does not apply
30
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
The negative statements follow the pattern as explained in the previous section. People tended to verbalize many statements that represented the path one was following and the direction one was going on the website. Moreover, it should be noted that people verbalized their expected results quite often, where not finding the expected result (36 times) occurred twice as often as finding the expected results (17 times). However, these were only the statements, which does not necessarily mean the result was not found. Additionally, people verbalized approximately as much positive evaluation statements as negative. Lastly, people highlighted positive aspects of the interface five times, which is half of the number of negative statements. A statement that was given multiple times, was that people found it pleasant that one could choose between searching via typing and searching by picking a topic from the list. Participants chose typing (55.45%) over picking a topic from the list (44.55%).
Participants switched search strategy, in total, 18.18% of the time within one task.
Although the occurrence of negative statements is considerable, the correctness of answer was 0.84, which means 84% of the tasks were completed within five minutes, representative of a good findability. Therefore, the perception was more negative than the actual performance.
Attitude and Findability
As explained, the concept findability was split into two parts: findability measured with performance metrics and the perceived findability of the participants that resulted from the think-aloud session.
Performance. For each performance metric, it was tested whether it was related to the attitude individuals have towards online health information. The means and standard deviations can be found in Table 4. Correlational analysis showed no significant relationship
31
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
between attitude and time in seconds ((22)= -.046, p = .774), error rate ((22)= .303, p =
.062), and correctness of answer ((22)= .172, p = .339). It is worth noting that the relationship between attitude and error rate is marginally significant and may be positively related.
Table 4
Means and standard deviations of findability for each sample Patient Non-patient Total
Time in 153.93 (26.31) 148.65 (29.94) 150.09 (28.48) seconds
Error rate 4.51 (1.14) 4.41 (1.60) 4.44 (1.46)
Correctness 0.93 (0.10) 0.80 (0.48) 0.84 (0.04) of answer
Perception. Statements from the participants regarding perceived findability were coded and split up into three broad groups for analysis, based on their sentiment: positive statements, negative statements and neutral statements. The means and standard deviations can be found in Table 5. Correlational analysis showed no significant relationship between attitude and negative statements ((22)= -.163, p = .335), positive statements ((22)= -.259, p
= .128), and neutral statements ((22)= .030, p = .859).
32
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Table 5 Means and standard deviations of findability for each sample Patient Non-patient Total
Negative 4.00 (2.97) 4.19 (2.11) 4.14 (2.30) statements
Positive 2.33 (2.73) 2.50 (1.32) 2.45 (1.74) statements
Neutral 2.33 (1.37) 2.88 (2.31) 2.73 (2.07) statements
Expertise
Performance. Correlational analysis showed no significant relationship between expertise and time in seconds ((22)= -.045, p = .775), error rate (r(22)= .100, p = .659), and correctness of answer (r(22)= .212, p = .344).
Perception. Correlational analysis showed no significant relationship between expertise and negative statements ((22)= -.234, p = .162), positive statements ((22)= -.029, p = .861), and neutral statements ((22)= .234, p = .167).
Illness Status
Performance. To test the differences between patients and non-patients regarding the performance metrics, multiple independent sample(s) t-test were performed. The means and standard deviations can be found in Table 4. The data was not normally distributed for time in seconds and correctness of answer. Therefore, the p-value may not be reliable and more weight should be placed on the bootstrapped 95% confidence interval that will be provided.
No significant difference was found regarding time (Mdif = 5.28, t(20) = 0.38, p = .708, 95%
33
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
CI (-20.84, 28.95)), error rate (Mdif = 0.11, t(20) = -0.15, p = .882, 95% CI (-1.39, 1.60)), and correctness of answer (Mdif = 0.13, t(20) = 1.59, p = .127, 95% CI (0.01, 0.25)).
Perception. To test the differences between patients and non-patients regarding their perception, multiple independent sample(s) t-test were performed. The means and standard deviations can be found in Table 5. The data was not normally distributed for negative and neutral statements. Therefore, the p-value may not be reliable and more weight should be placed on the bootstrapped 95% confidence interval that will be provided. No significant differences were found between patients and non-patients and their perception, nor negative statements (Mdif = -0.19, t(20) = -0.17, p = .869, 95% CI (-2.65, 2.54)), nor positive statements (Mdif = -0.17, t(20) = -0.14, p = .891, 95% CI (-2.20, 2.45)), nor neutral statements
(Mdif = -0.54, t(20) = -0.54, p = .598, 95% CI (-2.14, 0.95)).
Few users experiencing a problem (< 3) Many users experiencing a problem ( =/> 3) Patients (N = 6) Low severity Medium severity Small impact on findability Certainty about action planning (5)
Orientation (3)
Interface manipulation (3)
Patients (N = 6) Medium severity High severity Large impact on findability Verbalizing the preference for another Stuck in loops (3) website (1) Structure: evaluation (9)
Expected result (4) Figure 3. Findability issues for patients
34
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
As can be seen in Figure 3, patients mostly verbalized statements of affection for the organization of information or non-content features of the website, such as “Jeetje dit schiet ook niet op” (“Jeez, this is not going well/fast.”). The second most occurred category was
“Certainty about action planning”, which in the negative sentiment holds statements that represent being uncertain about where to click next. Moreover, some participants experienced the feeling of expecting a result, but not receiving it. Notably, content categories which hold statements about lack of information given about certain specialisms or the use of jargon were not found in the transcriptions of the patients. Additionally, no categories occurred in the ‘low severity’ box, which means all negative statements that were given by the patients had a medium or high severity.
(N = 16) Few users experiencing a problem (<5) Many users experiencing a problem ( =/> 5) Small impact on Low severity Medium severity findability Certainty about action planning (1) Interface manipulation (6)
Orientation (3) Large impact on Medium severity High severity findability Stuck in loops (4) Expected result (22)
Use of jargon (2) Verbalizing the preference for another website (5) Lack of information given about specialism (4) Not knowing which search topics to use (10)
Structure: evaluation (5)
Figure 4. Findability issues for non-patients
Interestingly, non-patients seem to experience more problems concerning the information that is provided by the website (see Figure 4). More specifically, non-patients
35
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
experience problems regarding the use of jargon, lack of explanation about specialisms (e.g.,
What is a haptotherapist?), and with quick searching because one did not know with which words to continue searching. Moreover, non-patients felt the need to continue searching on other websites, like Google (e.g., “Wait, I will Google this.”) more often than patients.
The frequency of statements within each category is provided in Table 6. Relatively seen, the data of both groups follow a similar pattern. Only the last categories (Use of Jargon,
Lack of Information Given about Certain Specialisms, Search Topics, and Preferences for other sites) occurred more often by non-patients, as discussed earlier in the issue analysis.
In short, the biggest difference between patients and non-patients lies in the last few categories, and especially in the categories involving the understanding of the content (i.e., information) of the website.
36
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Table 6
Frequency of statements for both groups Patients (N = 6) Non-patients (N = 16) Negative Positive Neutral Negative Positive Neutral
Subcategory statements statements statements statements statements statements
Certainty about 5 1 3 1 5 12 action planning Orientation 4 2 0 4 2 1 Expected result 6 5 * 30 12 * Reasoning about navigation logic 0 0 2 0 4 15 Interface 3 0 0 7 5 1 manipulation Stuck in loops 3 * * 4 * * Structure: 9 6 1 6 10 2 evaluation Structure: * * 8 * * 21 maintenance Use of jargon 0 * * 2 * *
Lack of information given about specialism 0 * * 4 * * Search topics 0 * * 10 * *
Preference for 1 * * 5 * * other sites Note. * does not apply
37
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Discussion and Conclusion
The purpose of the current study was to determine the findability of information of cancer information-related websites (main RQ) and to ascertain whether attitudes towards online health information (RQ1), expertise in searching for online health information (RQ2) and illness status (RQ3) are determinants for findability of information. To achieve the aims,
6 patients and 16 non-patients participated in a survey and a think-aloud session.
The main research question aimed to answer what the findability of information of
IKNL’s website http://www.kanker.nl/verwijsgids was. Prior studies showed that in general, giving up on information searches can be divided into three parts: technical problems (e.g., time to download web pages), the lack of information on a website, and bad usability of a website (Schenkman, & Jönsson, 2000, Chevalier, & Kicka, 2006; Chevalier, Maury, &
Fouquerau, 2014). All three aspects were witnessed during the scenarios, yet still led to finding the answer within five minutes in 84% of the cases, thus not giving up on an information search. Participants experienced a lack of information given on the website about certain specialisms (e.g., haptotherapist). This was mostly the case for non-patients. Negative statements regarding findability included problems such as different information given when searching via different ways for the same topic, loops within the website, and a footer that hindered reading the text properly. Moreover, some participants experienced lostness, which hindered findability. However, positive statements about the usability were also present.
Participants appreciated that there were multiple options to start searching and one had the ability to pick a topic from the list. Moreover, it was appreciated that one could filter the information with given filters. A possible explanation for not giving up the information search may be caused by the context in which the searches took place. More specifically, participants
38
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
were conscious one was participating in research. If the problems occurred in a situation with a higher ecological validity, one may have given up their search faster.
Categories involving content features included almost only negative statements from non- patients, but not from patients. A possible explanation for this could be health literacy. People with higher health literacy skills are better at applying literacy skills to health-related materials, such as directions for home health care (Parker, Baker, Williams, & Nurss, 1995).
It is presumable patients and relatives are more trained and learned in this than non-patients.
In this study, all participants verbalized at least one negative statement about the website, which represents experiencing problems in finding the information. Furthermore, more negative statements were verbalized than neutral or positive statements. The problems vary from not being able to tick a box to use filters to not understanding the given information.
Although every participant experienced problems, with either low, medium, or high impact on findability, most of the tasks were completed within the given times.
A prior study that looked at the findability of information included 200 breast cancer patients. Results revealed that 31% of the patients who used the internet to gain information about breast cancer experienced problems in finding the information they were looking for
(Littlechild, & Barr, 2013). Results of the prior study and the current study differ, but this could be the result of the operationalization of findability. To specify, the current study made a distinction between the performance regarding findability and the perception. When looking at performance, findability is regarded more positively than when looking at the data of the perception. Therefore, percentages can differ due to the operationalisation of findability.
Another possible explanation for the mismatch could be that people verbalize negative statements sooner because it is more striking when something is not working or not easy to find than when it is.
39
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
In sum, the findability of the website of IKNL differs when looking at performance and perception, but is overall of sufficient quality to find information. However, some usability issues were revealed and should be revised to improve the findability of information and thereby contribute to several benefits for cancer patients.
The first research question was established to find out if attitude towards online health information is a determinant for findability of information. Prior studies showed that in general, cancer survivors have a positive attitude towards online health information. (Jansen et al., 2015). In this study, the average attitude towards online health information confirms this finding. This study was a first attempt to test the relationship between attitude towards online health information and findability of information. Multiple correlation analysis revealed that there is no significant relationship between attitude towards online health information and findability (RQ1).
The second research question was established to find out if expertise in searching for online health information is a determinant for findability of information. Among others,
Potosky (2006) stated that unfamiliarity with the internet can be a disadvantage when using the internet in comparison with people that are more knowledgeable about the internet and its usage. To test this statement specified to findability of information on the internet, a correlational analysis was conducted. No significant relationship between expertise in searching for online health information and findability was found (RQ2), which suggests that more experience with using the internet and being more knowledgeable about the internet and its usage seems to have no direct influence on the findability of information. The results of both studies may differ because the current study made use of self-reported expertise, which was tested before the tasks were completed. Therefore, people may have underestimated themselves or were cautious because one did not know yet how one would perform.
40
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Lastly, the third research question was established to find out whether illness status (i.e., patient vs. non-patient) is a determinant for findability of information. A study from Xiao et al. (2014) showed that cancer patients tend to seek cancer information more intensively than healthy people (i.e., non-patients). To test if this determined findability of information, multiple independent t-tests were performed comparing patients and non-patients. For both performance and the perception of findability, no significant differences were found, which indicates that illness status, and thereby searching more intensively as a patient, is not a determinant for findability of information.
In conclusion, the results of this study indicate that neither attitude towards online health information, expertise in searching for online health information, nor illness status are determinants of findability of information (RQ1, RQ2, RQ3).
Limitations and Implications for Further Research
Although various valuable practical implications for IKNL’s website can be given, this study has some limitations as well. Firstly, the research design is limited. The survey was split up into two parts: the first part with general statements about the attitude towards online health information was filled out before consulting the website (i.e., before the think-aloud session). The second part included more website specific statements and was filled out after consulting the website during the think-aloud session. Therefore, the survey may be biased due to the think-aloud session. However, for the second part, it was necessary to have seen the website. If this was done before the first part, this would, in turn, have improved the participants’ knowledge of the website and could influence the task performance during the think-aloud method. In sum, results can be influenced due to the order of methods.
Another limitation concerning the order was that the order of the scenarios was not randomized. This decision was made because the first and last question had to be the least
41
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
heavy and difficult. Therefore, it may be possible that due to a learning curve, the last scenarios were done more quickly and more correct. Although the results do not give an indication for this, all scenarios were analysed together and overall means were used for the analysis.
As explained in the method section, the survey consisted, among other things, of the
Dutch version of e-Health Impact Questionnaire (e-HIQ-NL), which is currently being validated by PhD-student Koen Neijenhuijs and professor doctor Irma Verdonck-de Leeuw.
After the current study was conducted, the validation analysis brought forward some deviations from the original questionnaire. More specifically, another factor structure comes forward from the analysis, which technically means other subscales are at hand. However, these are provisional results and should be taken with caution. The subscale ‘Attitude towards online health information’ that was used for the analysis of the variable attitude, is therefore still included with its original items (in Dutch). The Cronbach’s alpha for this scale was marginally acceptable.
A possible explanation for the non-significance of all the statistical tests could be the sample size (N = 22 for total sample; N = 6 for patients and relatives sample; N = 16 for student sample). These small sample sizes could have much ‘noise’ and therefore it is more difficult to find the ‘signal’. Moreover, in small samples, large differences can be non- significant (Field, 2013). The same tests could, therefore, produce significant results in a larger sample with the same measurements. A follow-up study with a larger sample could be done to account for this possible explanation.
This study did not take all possible user characteristics into account and there may be user characteristics that determine findability of information. For example, a study by
Littlechild and Barr (2013) found that non-white patients (64%) experienced more problems
42
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
than white patients (25%) (Littlechild, & Barr, 2013). A possible research question for future research is: “To what extent does ethnicity play a role in findability of information?” This prior study discussed whether the language used on the website was the first language of the participant or not as a possible explanation for their finding. Therefore, another question that could yield new insights is: “To what extent does language play a role in findability of information?”
Theoretical and Practical Implications
Taken together, the findings suggest that usability tests regarding findability of information on online health information related websites could make use of non-patients, who are not directly representative end users of the websites. Non-patients are an easier group to include for sampling than most patients. However, further work needs to be performed to establish whether this finding of no difference holds for larger sample sizes.
Findings of these studies suggest the findability of information was mostly hindered by usability problems and not by user characteristics. These findings imply that evaluation of websites is important to yield new insights about findability issues and improve the website.
Moreover, a good findability can have great benefits for cancer patients, such as less anxiety and depression (Leydon et al., 2010). Results showed that websites do not need to be specified towards cancer patients and that focussing on attitude towards online health information or enhancing expertise in searching for online health information is not necessary to improve the findability. In sum, the focus of chance should mostly lay on the website side, and not on the user’s side.
This research has several practical applications. Firstly, it points to the search strategy, which shows that people like to have multiple search options: namely typing one’s own search words and picking topics from a list. Moreover, loops are found to be very confusing
43
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
and lower the perceived findability of information. In addition, if people have tried multiple times to find a certain piece of information on a website, and this fails, they are disposed to go to another website. In this case, another health information website could provide good information as well, but it is unfavorable if people go to the competitor to find information.
Most of these issues are not unique to the website used in this study but can occur in other websites as well. Most important is to evaluate websites and to find the underlying issues that can hinder the findability. In that way, users can experience benefits from finding the information and as the company can benefit from a good website as it may lead to more users.
Practical Implications for IKNL Specific
Besides general practical implications, some implications are mostly relevant for
IKNL’s website http://www.kanker.nl/verwijsgids. These implications are summed up in this section and are communicated more elaborated to IKNL. Firstly, staff members of IKNL acknowledged that there were doubts concerning giving visitors of these website multiple ways to search for certain information. This study showed that people use both ways (i.e., searching via typing and searching via picking a topic), so these should both be retained.
However, searching for the same information, through another means (for example typing instead of picking a topic from the list) can give people other information and other filter options. This is confusing and should be made consistent. Moreover, more information about certain specialisms could be added, to help people with lower health literacy. In addition, more filter options could be added, to make the search more specific and to preclude people having to search through many pages full of search results, and then dropping out because of it.
44
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
References
Ajzen, I. (1985). From Intentions to Actions: A Theory of Planned Behavior. In: Kuhl J.,
Beckmann J. (eds) Action Control. SSSP Springer Series in Social Psychology.
Springer, Berlin, Heidelberg
Cambridge Dictionary. (n.d.). attitude Meaning in the Cambridge English Dictionary.
Retrieved from https://dictionary.cambridge.org/dictionary/english/attitude
Cambridge Dictionary. (n.d.). expertise Meaning in the Cambridge English Dictionary.
Retrieved from https://dictionary.cambridge.org/dictionary/english/expertise
Castleton, K., Fong, T., Wang-Gillam, A., Waqar, M., Jeffe, D., Kehlenbrink, L., Gao, F.,
Govindan, R (2011). A survey of internet utilization among patients with
cancer. Supportive Care in Cancer : Official Journal of the Multinational Association
of Supportive Care in Cancer, 19(8), 1183-90. doi:10.1007/s00520-010-0935-5
Chevalier, A., & Kicka, M. (2006). Web designers and web users: Influence of the ergonomic
quality of the web site on the information search. International Journal of Human -
Computer Studies, 64(10), 1031-1048. doi:10.1016/j.ijhcs.2006.06.002
Chevalier, A., Maury, A., & Fouquereau, N., (2014) The influence of the search complexity
and the familiarity with the website on the subjective appraisal of aesthetics, mental
effort and usability, In Behaviour & Information Technology, 33:2, 117-132, DOI:
10.1080/0144929X.2013.819936
Dhyani, D., Keong Ng, W., & Bhowmick, S. (2002). A survey of web metrics. Acm
Computing Surveys, 34(4), 469-503. doi:10.1145/592642.592645
45
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Elm, W. C., & Woods, D. D. (1985). Getting Lost: A Case Study in Interface
Design. Proceedings of the Human Factors Society Annual Meeting, 29(10), 927-929.
doi:10.1177/154193128502901006
Eveland Jr, W. P., & Dunwoody, S. (2000). Examining information processing on the World
Wide Web using think-aloud protocols. Media Psychology, 2(3), 219-244.
Eysenbach, G., Powell, J., Kuss, O., Sa, E. (2002). Empirical Studies Assessing the Quality of
Health Information for Consumers on the World Wide Web: A Systematic Review. In
JAMA;287(20):2691–2700. doi:10.1001/jama.287.20.2691
Eysenbach, G., (2003). The impact of the internet on cancer outcomes. Ca: A Cancer Journal
for Clinicians, 53(6), 356-71.
Field, A. (2013). Discovering statistics using iBM sPSS statistics : And sex and drugs and
rock 'n' roll(4th edition. ed.). Los Angeles: Sage.
Grigoroudis, E., Litos, C., Moustakis, V., Politis, Y., & Tsironis, L. (2008). The assessment of
user-perceived web quality: Application of a satisfaction benchmarking
approach. European Journal of Operational Research, 187(3), 1346-1357.
doi:10.1016/j.ejor.2006.09.017
IKNL. (2010). about IKNL. Retrieved from https://www.iknl.nl/over-iknl/about-iknl
Jansen, F., Van Uden-Kraan, C., Van Zwieten, V., Witte, B., & Verdonck-de Leeuw, I.
(2015). Cancer survivors’ perceived need for supportive care and their attitude
towards self-management and eHealth. Supportive Care in Cancer, 23(6), 1679-1688.
doi:10.1007/s00520-014-2514-7
46
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Jaspers, M.W.M., Steen, T., Bos, van den C., & Geenen, M. (2004). The think-aloud method:
a guide to user interface design. International Journal of Medical Informatics, 73(11-
12), 781-795.
Khazaal, Y., Chatton, A., Cochand, S., Coquard, O., Fernandez, S., Khan, R., … Zullino, D.
(2009). Brief DISCERN, six questions for the evaluation of evidence-based content of
health-related websites. Patient Education and Counseling, 77(1), 33-37.
doi:10.1016/j.pec.2009.02.016
Krahmer, E., & Ummelen, N. (2004). Thinking about thinking aloud: A comparison of two
verbal protocols for usability testing. Ieee Transactions on Professional
Communication Vol.47 (2004) Nr.2 P.105-117 [issn 0361-1434]
Kim, D., & Chang, H. (2007). Key functional characteristics in designing and operating health
information websites for user satisfaction: An application of the extended technology
acceptance model. International Journal of Medical Informatics, 76(11), 790-800.
doi:10.1016/j.ijmedinf.2006.09.001
KWF Kankerbestrijding. (n.d.). Leven na kanker, 3 ervaringen - KWF Kankerbestrijding.
Retrieved from https://www.kwf.nl/kanker/gevolgen-van-kanker/pages/default.aspx
LaCoursiere, S.P., Knobf, M.T., & McCorkle, R. (2005). Cancer patients' self-reported
attitudes about the internet. Journal of Medical Internet Research, 7(3), 22.
Littlechild, S., Barr, L., (2013). Using the internet for information about breast cancer: A
questionnaire-based study. Patient Education and Counseling, 92(3), 413-417.
doi:10.1016/j.pec.2013.06.018
Leydon, G., Boulton, M., Moynihan, C., Jones, A., Mossman, J., Boudioni, M., &
McPherson, K. (2000). Cancer Patients' Information Needs and Information Seeking
47
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Behaviour: In Depth Interview Study. BMJ: British Medical Journal, 320(7239), 909-
913. Retrieved from http://www.jstor.org/stable/25187574
McMullan, M. (2006). Patients using the internet to obtain health information: How this
affects the patient-health professional relationship. Patient Education and
Counseling, 63(1), 24-28. doi:10.1016/j.pec.2005.10.006
Mayer, D., Terrin, N., Kreps, G., Menon, U., McCance, K., Parsons, S., & Mooney, K.
(2007). Cancer survivors information seeking behaviors: A comparison of survivors
who do and do not seek information about cancer. Patient Education and
Counseling, 65(3), 342-350. doi:10.1016/j.pec.2006.08.015
Mokkink, L. B., Terwee, C. B., Patrick, D. L., Alonso, J., Stratford, P. W., Knol, D. L., ... &
De Vet, H. C. (2009). The COSMIN checklist manual. Amsterdam: VU University
Medical Centre.
Morahan-Martin, J.M., (2004). How internet users find, evaluate, and use online health
information: A cross-cultural review. Cyberpsychology & Behavior : The Impact of
the Internet, Multimedia and Virtual Reality on Behavior and Society,7(5), 497-510.
Moreno, J., Morales del Castillo, J., Porcel, C., & Herrera-Viedma, E. (2010). A quality
evaluation methodology for health-related websites based on a 2-tuple fuzzy linguistic
approach. Soft Computing : A Fusion of Foundations, Methodologies and
Applications,14(8), 887-897. doi:10.1007/s00500-009-0472-7
Morville, P. (2005). Lost and found. In Ambient findability (p. 4). Sebastopol, CA: O'Reilly.
48
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Nagler, R., Gray, S., Romantan, A., Kelly, B., DeMichele, A., Armstrong, K., Hornik, R.
(2010). Differences in information seeking among breast, prostate, and colorectal
cancer patients: Results from a population-based survey. Patient Education and
Counseling, 81(1), 5-5.
Newnham, G., Burns, W., Snyder, R., Dowling, A., Ranieri, N., Gray, E., & McLachlan, S.
(2006). Information from the internet: Attitudes of australian oncology
patients. Internal Medicine Journal, 36(11), 718-723. doi:10.1111/j.1445-
5994.2006.01212.x
Nielsen, J. (1993). Usability engineering ([Rev. ed.]. ed.). San Diego, CA etc.: Morgan
Kaufmann.
Nielsen Norman Group. (2014, January 12). Task Scenarios for Usability Testing. Retrieved
from https://www.nngroup.com/articles/task-scenarios-usability-testing/
Otter, M., & Johnson, H. (2000). Lost in hyperspace: Metrics and mental models. Interacting
with Computers, 13(1), 1-40. doi:10.1016/S0953-5438(00)00030-8
Oxford University Innovation. (n.d.). e-Health Impact Questionnaire (eHIQ) - Oxford
University Innovation. Retrieved from https://innovation.ox.ac.uk/outcome-
measures/e-health-impact-questionnaire-ehiq/
Parker, R., Baker, D., Williams, M., & Nurss, J. (1995). The test of functional health literacy
in adults. Journal of General Internal Medicine, 10(10), 537-541.
doi:10.1007/BF02640361
Potosky, D. (2007). The internet knowledge (iKnow) measure. Computers in Human
Behavior, 23(6), 2760-2777. doi:10.1016/j.chb.2006.05.003
49
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Quesenbery, W. (2004). Balancing the 5Es of usability. Cutter It Journal, 17, 4-11.
Rugg, G., & Petre, M. (2006). Gentle guide to research methods. Retrieved from
http://ebookcentral.proquest.com
Schenkman, B., & Jönsson, F. (2000). Aesthetics and preferences of web pages. Behaviour &
Information Technology, 19(5), 367-377. doi:10.1080/014492900750000063
Simunich, B., Robins, D. B., & Kelly, V. (2015). The impact of findability on student
motivation, self-efficacy, and perceptions of online course quality. American
Journal of Distance Education, 29(3), 174-185.
Treadwell, D. (2017). Surveys. In Introducing Communication Research (3rd ed., pp. 158-
159). Thousand Oaks, CA: SAGE Publications.
Tullis, T., & Albert, B. (2013). Measuring the user experience : Collecting, analyzing, and
presenting usability metrics. Amsterdam: Elsevier.
Wyatt, J.C., & Liu, J.L.Y. (2002). Basic concepts in medical informatics. In Epidemiol
Community Health 56: 808-812. doi: 10.1136/jech.56.11.808
Xiao, N., Sharman, R., Rao, H.R., & Upadhyaya, S. (2014). Factors influencing online health
information search: An empirical analysis of a national cancer-related
survey. Decision Support Systems, 57(1), 417-427. doi:10.1016/j.dss.2012.10.047
50
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Appendices
Appendix A
Survey
All items are measured on a five point Likert Scale, items ranging from “Totally Disagree (1) to Totally Agree (5)”.
Part one
Attitudes towards online health information
1. Het internet is een betrouwbare bron om mij te helpen begrijpen wat een arts mij
vertelt.
2. Het internet kan mensen helpen om te weten hoe het is om te leven met een
gezondheidsprobleem.
3. Het internet kan nuttig zijn om mensen te helpen beslissen of hun symptomen
belangrijk genoeg zijn om een arts te raadplegen.
4. Ik zou het internet gebruiken als ik hulp nodig zou hebben bij het maken van een
beslissing over mijn gezondheid (bijvoorbeeld of ik een arts zou moeten raadplegen,
medicatie zou moeten innemen of andere typen behandelingen zou moeten zoeken).
5. Ik zou het internet gebruiken om na te gaan of de arts mij passend advies geeft.
Attitudes towards online sharing of health experiences
6. Het internet is een geode manier om andere mensen te vinden die vergelijkbare
gezondheidsproblemen ervan.
7. Het kan behulpzaam zijn om gezondheids-gerateerde ervaringen van andere mensen
op het internet te zien.
51
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
8. Het internet is nuttig als je niet wilt vertellen aan mensen in je omgeving (bijvoorbeeld
uw familie of collega’s) hoe je je voelt.
9. Het kan geruststellend zijn om te weten dat ik op elk moment van de dag of nacht
terecht kan op gezondheidsgerelateerde websites.
10. Het internet is een goede manier om andere mensen te vinden die geconfronteerd zijn
met gezondheidsgerelateerde beslissingen waar ik mogelijk ook mee wordt
geconfronteerd.
11. Het bekijken van websites over gezondheid stelt me gerust dat ik niet alleen ben met
mijn gezondheidszorgen.
Expertise in searching for online health information
12. Wanneer ik online zoek naar gezondheidsinformatie, vind ik snel wat ik zoek.
13. Als ik online zoek naar gezondheidsgerelateerde informatie dan vind ik normaal
gesproken succesvol de informatie die ik zoek.
14. Ik surf snel op het internet.
15. Ik krijg actuele gezondheidsinformatie van het internet.
16. Ik heb eerder gezocht naar gezondheidsgerelateerde websites op het internet.
17. Ik klik vaak op links die mij leiden naar nieuwe websites wanneer ik zoek naar
gezondheidsinformatie op het internet.
Part 2
Trust and identification
18. De website bereidt me voor op wat er mogelijk gaat gebeuren met mijn gezondheid.
19. De mensen die hebben bijgedragen aan de website begrijpen wat voor mij belangrijk
is.
20. Ik heb een gevoel van solidariteit met andere mensen die de website gebruiken.
52
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
21. Ik kan me identificeren met andere mensen die de website gebruiken.
22. Ik waardeer het advies dat gegeven wordt op de website.
23. De website geeft me het vertrouwen dat ik in staat ben om met mijn gezondheid om te
gaan.
24. Ik heb het gevoel veel gemeen te hebben met andere mensen die de website gebruiken.
25. De website geeft mij het vertrouwen om mijn gezondheid te bespreken met mensen in
mijn omgeving (bijvoorbeeld mijn familie of collega’s).
26. De website geeft mij meer vertrouwen om mijn gezondheid te bespreken met mensen
in mijn omgeving (bijvoorbeeld mijn familie of collega’s).
Information and presentation
27. De informatie op de website liet een gevoel van verwarring bij me achter.
28. De website biedt een breed scala aan informatie.
29. De taal op de website maakte het gemakkelijk te begrijpen.
30. Ik kan de informatie op de website gemakkelijk begrijpen.
31. Ik vertrouw de informatie op de website.
32. Foto’s en andere afbeeldingen op de website werden passend gebruikt.
33. Ik vond de afbeeldingen op de website verontrustend.
34. De website is gemakkelijk te gebruiken.
Understanding and motivation
35. De website moedigt mij aan om acties te ondernemen die gunstig kunnen zijn voor
mijn gezondheid.
36. De website heeft een positieve kijk.
37. De website bevat nuttige tips over hoe het leven beter te maken.
38. Ik voel me meer geneigd om op mezelf te letten na het bezoeken van de website.
53
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
39. Ik heb iets nieuws geleerd van de website.
40. Ik zou de website raadplegen als ik een beslissing zou moeten nemen over mijn
gezondheid.
41. In zijn geheel, vind ik de website geruststellend.
42. De website helpt me begrip te hebben van mijn persoonlijke gezondheid
43. De website moedigt mij aan om een actievere rol te spelen in mijn gezondheidszorg.
Demographic questions
44. Wat is uw leeftijd? – Open vraag
45. Wat is uw geslacht? – Man / Vrouw
46. Wat is uw hoogst genoten opleiding? – mbo/hbo/wo
47. Heeft de website waar zojuist over gesproken is al vaker bezocht? – Nee nooit / Ja een
enkele keer / Ja regelmatig
48. Mijn relatie tot kanker: (Ex-)patiënt / Familie / Nabestaande / Persoon met erfelijk
risico / Anders / Student Tilburg University
49. Kankersoort - Open vraag
54
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Appendix B
Scenarios and tasks
Scenarios and tasks 1. Person A is has lost much of her energy and deals with fatigue. Person A heard that a course is made for this, which is called ‘Minder moe bij Kanker’. Search which organisation offers this course and find contact information for person A so she can gather more information about the course (for example an email adress).
Persoon A is na chemotherapie veel van haar energie kwijtgeraakt en kampt dan ook met vermoeidheid. Persoon A heeft vernomen dat hier een cursus voor is ontwikkeld, namelijk ‘Minder moe bij Kanker’. Zoek welke organisatie deze cursus aanbiedt en waar persoon A meer informatie kan verkrijgen (bijvoorbeeld via een e-mailadres of website). 2. Person B is planning on visiting a sexologist, but is looking for some additional information. More specifically, this person is not sure if a reference form from the doctor is needed and if the costs will be reimbursed. When you have found the answer to the question if a reference form is needed and if the costs will be reimbursed, write in key words what information is distributed by the Verwijsgids for these questions.
Persoon B is van plan om een seksuoloog te bezoeken, maar is nog op zoek naar wat aanvullende informatie. Deze persoon weet niet zeker of er een verwijsbrief van de dokter nodig is en of de kosten van de seksuoloog worden vergoed. Zoek het antwoord op deze punten, dus: of er een verwijsbrief nodig is én of de kosten vergoed zullen worden en schrijf in kernwoorden op wat hierover gezegd wordt op de site. 3. Person C experiences feeling of loneliness after her breast cancer treatment. She would like to have contact with peers. Person C has no idea how she can accomplish this wish and hopes to find more information in the Verwijsgids. She prefers to not leave her house and therefore wishes to have online peer contact. Moreover, she thinks it is important to have contact with peers who’ve dealt with the same kind of cancer, namely breast cancer. Look for an organisation that suits her wishes and write down an mail adress person C can send an email to, to retrieve further information.
55
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Persoon C is na een borstkankerbehandeling eenzaam geworden en zou graag in contact komen met lotgenoten. Deze persoon heeft echter geen idee hoe dat zou kunnen worden bewerkstelligt en hoop hiervoor in de verwijsgids een oplossing te vinden. Het liefst wil deze persoon hiervoor niet de deur uit en heeft dan ook de voorkeur voor online lotgenotencontact. Daarnaast vindt ze het belangrijk dat de lotgenoten dezelfde kankersoort hebben ervaren, namelijk borstkanker. Ga op zoek naar een organisatie die hierbij aansluit en schrijf op naar welk emailadres persoon C een email kan sturen voor verdere informatie. 4. Person D has to deal with pain, which is not specifically situated in one part of the body. This person lives in Utrecht and would like to get help nearby. If necessary, person D is able to go 50 kilometres for the caregivers, but not further. Considering the following requirements: 1. Pain is not situated in one specific part in the body, 2. The person will not go further than 50 kilometres, which results given by the Verwijsgids suits the best? Write down the name of the practitioner, what kind of help (for example physiotherapy) it involves, and how far it is away from Utrecht. Persoon D heeft last van pijn. Deze pijn bevindt zich niet op één specifieke plek in het lichaam. Persoon D woont in Utrecht en wil graag hulp waarvoor niet ver gereisd hoeft te worden. Als het nodig is kan persoon D tot 50 kilometer reizen om bij de zorgverlener te komen. Rekening houdend met deze vereisten: de pijn zit niet op één specifieke plek en de persoon wil niet verder reizen dan 50 kilometer, welk resultaat dat de Verwijsgids geeft past hier het beste bij? Schrijf de naam van de zorgverlener op, welke soort hulp het omhelst (bijvoorbeeld fysiotherapie) en hoe ver (in kilometers) het van Utrecht vandaan is. 5. Person E has anxiety due to cancer and would like to orientate for some support options. Person E has not developed a specific preference and would like to orientate with the help of the Verwijsgids. Look what the Verwijsgids offers for anxiety, and orientate yourself by using filters. Choose one possibility. from the filters ‘Zelf doen’ or ‘Online’. Persoon E heeft als kankerpatiënt bepaalde angsten ontwikkeld en wil zich oriënteren naar de ondersteuningsmogelijkheden hierin. Persoon E heeft nog niet een concrete voorkeur en wil zich via de Verwijsgids oriënteren. Zoek naar het aanbod voor ‘Angst’ en probeer door middel van het gebruiken van filters je te
56
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
oriënteren in de mogelijkheden. Kies vervolgens uit de categorie ‘Zelf Doen’ of uit de categorie ‘Online’ één mogelijkheid.
57
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Appendix C
Consent form
Ik geef aan vrijwillig deel te nemen aan een onderzoeksproject uitgevoerd door de masterstudent Carlijn Raijmakers van de Universiteit van Tilburg. Ik begrijp dat het project is ontworpen om informatie te verzamelen over het gebruik van de IKNL website www.kanker.nl/verwijsgids.
Mijn deelname aan deze studie is vrijwillig. Ik begrijp dat ik niet betaald zal worden voor mijn deelname. Ik kan de deelname op elk moment zonder consequenties opzeggen en stoppen. Als ik weiger om deel te nemen aan of mij terug te trekken uit de studie, zullen er geen gevolgen zijn.
Ik begrijp dat de participanten emotioneel kunnen worden en de onderwerpen die besproken worden lastige onderwerpen zijn om over te praten. Als ik mij op de een of andere manier ongemakkelijk voel tijdens de sessie, heb ik het recht om te weigeren een vraag te beantwoorden of de sessie te beëindigen.
Deelname houdt in dat er een vragenlijst wordt afgenomen en een hardop denksessie waarin u wordt gevraagd om een aantal taken uit te voeren op de websites volgens een scenario. De sessie duurt ongeveer 40 minuten. Scherm- en geluidsopnamen zullen gemaakt worden om later te analyseren. Als ik niet opgenomen wil worden, zal ik niet kunnen deelnemen aan de studie.
Ik begrijp dat de onderzoeker mij niet per naam zal identificeren in alle rapporten waarin van de informatie die uit deze sessie is verkregen gebruik wordt gemaakt en dat mijn vertrouwelijkheid als deelnemer in acht genomen wordt. Volgende toepassingen van de opnames en andere data zijn onderhevig aan standaard datagebruik beleid die de anonimiteit van individuen en instellingen beschermen.
Ik heb de uitleg gelezen en begrepen. Ik heb al mijn vragen beantwoord gekregen tot mijn voldoening en ik ga vrijwillig akkoord om deel te nemen aan deze studie.
58
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
______Handtekening Datum
______Naam onderzoeker Handtekening onderzoeker
Voor meer informatie, neem alstublieft contact op met:
Carlijn Raijmakers [email protected]
Consent form English
I volunteer to participate in a research project conducted by Carlijn Raijmakers Master student from Tilburg University. I understand that the project is designed to gather information about the use of IKNL’s website www.kanker.nl/verwijsgids.
My participation in this study is voluntary. I understand that I will not be paid for my participation. I may withdraw and discontinue participation at any time without penalty. If I decline to participate or withdraw from the study, there will be no consequences.
I understand that participants may get emotional and find the topics discussed difficult to talk about. If I feel uncomfortable in any way during the session, I have the right to decline to answer any question or to end the session.
Participation involves an questionnaire, and thinking aloud session where you are asked to perform some tasks on the websites according to a scenario. The session will last approximately 40 minutes. Screen and audio recordings will be made to analyze later. If I don't want to be recorded, I will not be able to participate in the study.
I understand that the researcher will not identify me by name in any reports using information obtained from this session, and that my confidentiality as a participant in this study will
59
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
remain secure. Subsequent uses of records and data will be subject to standard data use policies which protect the anonymity of individuals and institutions.
I have read and understand the explanation provided to me. I have had all my questions answered to my satisfaction, and I voluntarily agree to participate in this study.
______My Signature Date
______Researchers name Signature of the researcher
For further information, please contact:
Carlijn Raijmakers [email protected]
60
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Appendix D
Elaborate version of procedure
In the introduction the participants are notified of the structure of the process (survey part one, think-aloud session, survey part two), and it is briefly explained what the aim of the session is.
More specifically, the participant is told the think-aloud session is meant to gain knowledge about how cancer patients evaluate the website, and to eventually give recommendations to
IKNL for optimization of the website. To reassure the participant, they are told that there is no wrong or stupid answer or clumsy way in doing the tasks. They are told that they can drop out when desired, without any consequences. To make sure this is handled in the right way, participants are asked to sign the consent form. After that, the real data gathering can begin.
Example instruction: “Thank you for participating in this research session. I want to inform you about what is going to happen in this session. First we start with a short survey about your online health experiences. After the short survey we start with a few tasks you need to carry out on the website. Each task is explained within a scenario. The scenarios are about people with cancer. If at any point you think it is too difficult or you get emotional we can stop the task and take a break before we start with a new task, or we stop the session. Do not feel obligated to do things you do not completely feel happy about. During the tasks I ask you to think-aloud. I know that is a hard thing to do, that is why I will show you how I do it with a dictionary, and afterwards I have a fun game where you can practise thinking aloud beforehand. There are 5 scenarios. You should know that there are no wrong or stupid questions, or clumsy ways of doing the tasks. After the thinking aloud is done a second survey is handed to you about the website. You may fill it out, and after that we complete this session. Do you have any questions or comments about this session? Then I will hand you this
61
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
consent form. On this form you can read all the rights you have during this session. If you agree you can sign the form and we start the session.”
The first part of the survey will be handed out hard copy and the participant will fill this out individually. If there are questions regarding the statements, an answer will be given.
After that, the part containing the think-aloud session will begin. First, by finding a word in a dictionary, the researcher will illustrate the means of a think-aloud session. After that, the participant will experience this him/herself by practicing thinking aloud with the online version of the Tower of Hanoi (Krahmer, & Ummelen, 2004). The latter will be finished when the researcher is convinced of the understanding. To achieve this, and to enhance thinking aloud, several question can be asked: “What are you thinking now?”, “Why did you do that?”, “How would you start?”, “can you elaborate on that?”. Also reassuring sounds could be made to enhance thinking aloud: “uhuh” (Nielsen, 1993). The participants will be asked to express their opinion regarding the usefulness of the website, the structure, the options for searching within the website and the feelings they get when searching for information.
Subsequently, the participant will get two minutes to scroll through the website and is asked to verbalize their first expression by asking questions like “What is your first impression?” “What do you like about the website?” “Are there already aspects that could be improved in your opinion?” “Do you think the website is clearly structured?” Then, a hard copy version of five scenarios and tasks will be handed out. The researcher will start the recording program and the timer. The timer is used to measure the time a participant's spends on one task. During the tasks, no questions were answered. If questions were so urgent the ignorance of them would stop the process, it was tried to give a counter-question like “What will happen if you do it?” (Nielsen, 1993). If the participant cannot find the answer (i.e. the goal of the task) to a task, the participant may ‘give up’ after trying, or the participant will be
62
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
stopped after five minutes. If the participant did not find the answer, the researcher may ask questions like “Why do you think you did not find the answer?”, “What do you think could be improved on the website regarding this task?”. These questions are only necessary if the participant did not answer these questions automatically during the thinking aloud for this task. Between the tasks, participants may ask questions about practical issues, for example:
“Can I click on every button?” “Can I use other websites to find the answer?” This procedure will be repeated for the other four tasks. If the researcher is not convinced of the thinking aloud itself, it can ask questions that we also used in practicing with the thinking aloud session.
After the thinking aloud session, the researcher will ask if the participants has any contributions to the already given statements during the thinking aloud method. These are recorded as well.
Subsequently, the participant will be handed out the second part of the survey, which will be filled out individually. If there are questions regarding the statements, an answer will be given.
Lastly, the participants will be thanked for their help and a little presented will be handed to them.
63
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Appendix E
Codebook
The distinction is made on the three parts discussed by Rugg and Petre (2016), wherefrom two are at hand. 1.1 Categories derived from literature Navigation problems and information processing Category Description Example Code Additional code
Certainty about Subjects (do not) see “I see which option I could CN01 + Positive action planning where they could go go next to find the - Negative (click) next, answer..” = Neutral
or they see several possibilities but “I see books, I see a lot of haven’t got a clue things...but where should I about which one to go now? Can I click this at select (-) all? OK, let me see of something is clickable here.”
Orientation Subjects (do not ) “So, I am now in the topic CN02 + Positive understand where part, and I want to go to the - Negative they are or can (not) homepage, so I can go back = Neutral interpret their current now.. “ location in the
context of other locations in the “What does this mean? website; asking Where am I now? I guess I rhetorical questions should be in a completely about the structure of different part of the the site; synthesizing website?” the structure of site
Expected result Subjects expect a “…and then I click this and CN03 + Positive certain result after hope something - Negative clicking a link, happens…and here it is! = Neutral
64
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
and this result does This is what I was looking occur (+) for.”
but this result does “…and then I click this and not occur (-) hope something happens…but it doesn’t.”
Repetition of Subjects think they “Oh … yes…and now I CN04 + Positive actions remember how to should be able to select a - Negative navigate because title here because I just did = Neutral they feel they did that too. Yes that works.” that before; Making
connections to past experience with “Oh … yes…and now I other sites or other should be able to select a media formats, title here…. Oh no, apparently I can’t.”
and the repetition succeeds (+)
but the (assumed) repetition fails (-)
Reasoning about Subjects start “A clock that is ticking… CN05 + Positive navigation logic reasoning about why ah, perhaps that’s a time - Negative the interface makes machine. Let me try = Neutral them think they can that….” find certain
information behind a link “A clock that is ticking… ah, perhaps that’s a time machine. Let me try that….”
Interface Subject have “There’s a calendar here, CN06 + Positive manipulation verbalize positive but when I click this and - Negative words (+), or this happens… that is = Neutral verbalize problems (- easy.“ ) about handling
certain objects in the
65
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
interface (e.g. “There’s a calendar here, dragging a pointer but when I click it nothing over a calendar in happens… how can I do order to get to this?” information about a certain year)
Stuck in loops Subjects think they … hey, I’ve been here CN07 n/a move on to a before! **, again this different location in stupid page. the website, but appear to (repeatedly) return to where they came from
Structure: Stating the name of “to the figures, breast IPS01 n/a maintenance links before or as cancer, processing” they are traversed
Structure: Expressing affection “I find the site very clear, IPS03 + Positive evaluation for the organization and the search options are - Negative of information or nicely structured” = Neutral non-content features
of site. “I don’t like how unstructured the table is. I think it is chaotic”
CN = comment navigation IPC = Information Processing Content IPS = Information Processing Structure
66
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
1.2 Categories derived from data (will be supplemented while coding) Understanding of content
Problem Description Example Code Additional code Use of jargon Subject has trouble “Uh.. I do not know what NUC01 n/a with understanding incidence means so I do words that are used in not know what to do with the text this” Lack of Subject has trouble “Uh.. I do not know what a NUC02 n/a knowledge about with completing the haptotherapist does and the specialism task due to lack of site does not tell me so I do knowledge (and not know it is suitable for information) about the patient” certain specialisms NUC = Negative understanding content
Problems with search topics
Problem Description Example Code Additional code Search topics The subject does not “I do not what to search for PST n/a know where to search next.” for next or what
search words to use. This is independent “I absolutely have no from orientation and clue.” navigation logic.
67
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Preference for other sites
Problem Description Example Code Additional code Preference for The subject wants to “By now, I would have POS n/a other sites continue the search gone to Google.” via Google or another
website. “Can I search for the answer on another website?”
68
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Appendix F
Intercoder reliability
Table 7
Intercoder reliability
Subcategory Cohen’s Kappa Approximate Significance Certainty about action planning .848 .000 – negative Orientation - negative .643 .008 Expected result - negative 1 .000 Repition of actions - negative 1 * Reasoning about navigation 1 * logic - negative Interface manipulation - 1 * negative Stuck in loops .423 .101 Certainty about action planning 1 .000 –positive Orientation –positive 1 * Expected result –positive .659 .011 Repition of actions -positive 1 * Reasoning about navigation 1 * logic -positive Interface manipulation - 1 * positive Certainty about action planning .961 .000 –neutral Orientation – neutral 1 * Expected result – neutral 1 * Repition of actions - neutral 1 *
69
USER CHARACTERISTICS AS DETERMINANTS OF FINDABILITY
Reasoning about navigation 1 * logic - neutral Interface manipulation - 1 * neutral Structure: maintenance .762 .002 Structure: evaluation - neutral 1 .000 Structure: evaluation – positive .962 .000 Structure: evaluation – 1 .000 negative Use of jargon 1 .000 Lack of information given 1 * about specialism Other .571 .025 Note. (*) The significance level could not be calculated.
70